To be fair, the paper is about optimizing your test cases and being realistic about what you can accomplish with testing. It is a push toward more efficient testing rather than the lack of testing.
It's important for HN to keep its distance from tldr culture. Longer articles are ok, even if many of us won't read them. It's true of any post that most people won't read it.
Snap summaries make for reflexive reactions. That's the shallow kind of discussion we're hoping to avoid here. What we want is reflection, which is slower, takes more energy, and leads to more considered exchanges.
Scientific papers aim for a similar objective. They also recognize the importance of being able to summarize. Abstracts are a good idea, and don't necessarily mean people will make snap judgments.
There's a lot of long rubbish on the Internet. Being able to quickly distinguish it from good stuff is essential to using your time sensibly.
The abstracts are often written by the people who wrote the paper, a tldr often isn't, this allows at best inaccuracies in the summation and at worst complete bias which then results in a big argument completely unrelated to the source material.
Frankly if you aren't willing to read the content you simply shouldn't comment on it until you have.
HN is not just for commenting, it is also for reading other people's submissions and comments. So, granted, it's better not to write comments without reading the article first, but why should that mean that people who have read the article should not write a summary?
> If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away. They are producing no information for you — or at least very little information. The value of the information they produce may not be worth the expense of maintaining and running the
tests.
This seems.. entirely unreasonable to me. Just because some area of code is not touched frequently doesn't mean that we should throw out tests pertaining to that section of code. Can someone justify this to me? This seems like a _terrible_ mistake.
The context is trivial tests which only failed before the developer wrote the code and have never failed since. An example given is a method setting x to 5 and then testing that method. Because maintaining code has a cost, tests such as these should be removed.
Regression tests are mentioned later and are not what the author is addressing here.
I would argue that "producing information" is only one benefit of tests. Having worked with decent-sized legacy code bases, another big benefit of tests is preventing future loss of information. Deleting tests removes those guardrails. If what you write is successful, then it will become a legacy system, and your predecessors who inherit it (and ultimately your customers) will be glad you didn't delete tests.
I worked at a company that didn't employ any unit testing and they were terrified of updating their aging software stack because it was constantly breaking as is. I was doing greenfield work and was paranoid about losing data, so I had extensive test-coverage. I undertook multiple complex refactors of the codebase and never had a problem thanks to my unit test coverage. My code was never the cause of any major data loss and I found plenty of very serious bugs in the legacy codebase.
If you want to reduce your code mass, the number one thing you should do is look at the code paths that have not been reached in a year and consider throwing them away.
This would depend on the kind of software you're making and the potential for an unhandled error to cause injury, expense or damage, but yeah.
I would really like some system that can tell me how many times a line of production code has executed in the last year. For an ecommerce web app, I can't imagine how a line of code that hasn't run in the last two years would still need to be there.
My immediate thoughts exactly, so I reread to confirm the parent said "consider" removing the code. I think this is reasonable. While considering the removal of unused code, we should should keep the code that handles unlikely failure scenarios.
I don't get it either. In particular, if a test isn't failing, there is no reason to change it--i.e., no maintenance cost. Sometimes tests are written such that they test the implementation rather than the behavior (as is the case with egregious mocking), so changes to the implementation cause the test to report false negatives--but this is a matter of faulty testing practice adding unnecessary maintenance costs.
Provided you're following good unit testing practice, I can't imagine an organization whose best investment is removing excess unit tests.
Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring.
Tests take time to run, which slows the development loop.
Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation.
When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily.
"Has been passing for a long time" isn't necessarily a good heuristic for identifying these situations. I'm just saying there are lots of costs to having a passing test in the suite.
> Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring.
Only when refactoring the public interface, and only when tooling isn't capable of automating the refactoring. In other words, only when you're doing something complex enough that there's a high chance you'll break things and cause tests to fail!
Hopefully one will catch said test failures before checking in - note that this will skew any CI metrics. Just because they didn't fail on your CI infrastructure within the past year, doesn't mean they didn't fail within the past year.
> Tests take time to run, which slows the development loop.
This is, perhaps, a good reason to make those tests run e.g. only for your overnight integration tests. But removal? Nervously auditing and reviewing changes (both your own, and other's in code reviews) for possible breaks because of reduced confidence that your unit testing will catch things also slows down the development loop. As does the debugging sessions when that isn't enough.
> Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation.
This is, perhaps, a good reason to relegate less informative tests to another directory. Perhaps the same directory that houses regression tests that your fuzzer found.
> When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily.
The purist might cry that refactoring doesn't change (external) behavior. But I'll note that it's just as easy to delete unit tests when you're changing behavior, as it is to delete them ahead of time. Even easier! Bonus: Deferring their deletion until then makes the fact that you're actually changing behavior a lot more obvious and explicit.
Ahh, a fibbed a little: It might be harder to get those deletions approved by your code reviewer if it's not obvious that changing behavior is OK. In my work environments, this is a feature.
I also am not sure how you would identify this with confidence. Sure, I can see the tests haven't failed on CI. However, maybe the tests fail all the time while the tests are being run during development.
I can at least confirm with certainty that it sometimes is a terrible mistake.
I port things. One approach I sometimes use is to stub in noop implementations of platform specific things. I generally use #pragmas or the like to generate a warning to come back to (which I can always later crank up to an error to ensure I've gone back and implemented everything.) It's common that the original authors were also "kind enough" to leave some noop implementations of their own - although usually not with the helpful #pragmas.
End result: Thousands of failing unit tests, most of which were passing for more than a year. Some are innocuous as minor string formatting changes caused by minor libc differences - but even that can be concerning if you want consistent, reliable results that don't vary by platform and threaten to cause platform specific edge cases.
I've never seen a codebase where even 10% of the failures are that innocuous.
I agree that unit testing is a waste if the project requirements change quickly. Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies.
Unit tests only make sense for components of a system which are BOTH critical and stable.
I think that integration tests are often way more useful for most projects.
Unit tests can take ages to write and if the project requirements are changing constantly, then you keep having to update them and it slows down development and kills productivity - I've seen this happen many times in previous companies.
In some environments, refactorings to the code will also refactor the tests. Doing test first is only a win in the situation where you don't have to duplicate work, and only if management wouldn't otherwise give you the time to write tests after the fact. Unfortunately, the former is less common, and the latter is more common.
1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests. [System-level since context of how software was being used is important.]
2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
3. Except for No 2, if X has business value and you can use a system or unit test use a system test for X. [Context again.]
4. Design a test with more care than you design the code. [They need to do something meaningful rather than just be there as a metric.]
5. Turn most unit tests into assertions. [Assertions describe properties you code should always have before, during, or after executions. Just use assertions for such checks that instead of hiding them in tests.]
6.Throw away tests that haven’t failed in a year. [Controversial claim. Says they tell you nothing. I believe the author thinks the design and assertions should make the software right from the beginning with the tests telling you where you're screwing up.]
7. Testing can't replace good development: a high test failure rare suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth.
8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers performance. Rewarding coverage or some other meaningless metrics can lead to rapid architectural decay.
9. Be humble about what tests can achieve. Tests don't improve quality: developers do.
There's a lot of good points in there, though. Talks about combinatorial explosion where tests often don't even measure correctness. Talks about how maintenance burden goes up over just good system testing and assertions with minimal unit testing. Meantions Toyota Production System showed you keep human in the loop and let them do any analysis that requires brains while automating just the mundane stuff. Mentions how hardware engineers duing Design-for-Testing philosophy embed little probes in most of their blocks to catch violation of correctness conditions during testing. Kind of black box plus deep, white box. Says that systems could similarly be wired so the system tests would set off alarms when they should. Just a few gems I saw.
I agree a shorter version of this essay would be beneficial.
When doing "dynamic" programming (Java + Spring + Maven, Python, anything JavaScript), I feel compelled to write more unit tests. Because the sand shifts under my feet and my crap breaks in surprising ways.
I despise most modern development, where entropy continues to creep in and nothing stays done.
Maybe more assertions would help mitigate my feelings of hopelessness and despair.
I tried to find you a decent intro to the best approach at this, Design-by-Contract. The Eiffel site has a nice, short intro with examples and benefits:
It has been implemented in Java with the stuff inside JavaDocs. I know JML/Krakatoa integrated it with provers. Simplest is using it for interface checks, though, of expectations of what component is doing vs what it's actually doing. The DbC style lets you document your assumptions, have formal properties to derive tests from, work right there with the code itself, and even automatically generate tests in some tools. That last one is huge benefit as you know the tests will always be relevant to the module and don't have to maintain them. You can do your own tests at the system-level on stuff the module-level contracts might not capture.
I'm not sure I can help you for most of the others. Remember, though, you can implement the DbC assertions in code that's in the modules themselves called before and after they run. These can be set to turn off with a flag during production compiles. You can also write generators that pull them out of comments in your source files then produce a version of your program with the checks in them as tests. I can't find my original articles post-PC crash but the one below builds in Ruby a generator for C language DbC:
For the most part I find myself writing unit tests to save time debugging. With a unit test I can create a limited context for my module to execute within instead of having to manually run through through several steps in the larger application to test out some piece of functionality.
In my previous job I was maintaining a big project which originally had zero unit-tests, it was apparently working well but once I started adding TDD I found several security issues, edge cases, and memory overflows; by the end of my contract I had written more than +2,800 unit and integration tests and the code coverage was still low at around 60% but I am very confident that new developers are going to take ownership of the project from the beginning without much training just because I wrote those tests. I consider testing an useful way to allow new contributors to join the development process because they will not be afraid to touch the code because the tests will throw a warning when something breaks after any modification, without tests they would surely have to spend more time checking the details of their modifications more than integrating into the team.
Sorry for the very literal answer. I honestly thought you were interested in how the coverage was calculated.
As an aside, I've worked on projects where coverage wasn't used and the rule was to "test the important bits" and I've also been on projects where coverage had to be >= ~98%. I wonder if a middle-ground approach would be effective.
I hate it when people write tests to cover every single function (to cover every line of code) but they pay almost no attention to the range of inputs/arguments passed to those functions.
I think that most bugs in production can be traced back to "An unexpected argument which was passed to a complicated function/method" - According to this, you should focus your testing efforts on methods/functions that have complicated logic and you should test them with a range of unusual arguments/inputs.
Writing tests for EVERY function/method using only obvious (common use-case) arguments is a waste of time.
The biggest problem with unit tests is they are abused as a metric for code quality. Pretty code coverage graphs and percent numbers are easy to present to managers. And outsource firms love these because it adds pointless work they can justify as "code quality".
But what happens is the unit tests become filled completely useless logic. Like a test with only mock objects that just tests if a method can be called. Useless but good for padding those code coverage numbers. And ironically it greatly decreases code quality. Because when there's these useless unit tests covering everything, it makes refactoring very difficult. And so no one refactors to improve the code because its too much work.
Care to provide any details on how this was already solved? I'd love to know and I'm sure the many people here who write tools for developers would also be interested.
Care to provide any details on how this was already solved?
Our standard refactoring tool, the original Refactoring Browser (RB), covered Unit Tests just the same as any other code. If you fired off a "canned" refactoring in it, it was guaranteed to be correct, you could multiply undo/redo it, and it covered everything currently loaded in the image, which typically included Unit Tests.
I notice in Visual Studio and also with CLion and some other "refactoring" tools for C++ I've played around with, that some "refactorings" amount to: "We'll generate/move code for you, but you're on your own with regards to correctness." This reduces the usability and productivity of the tool considerably.
In addition, the Refactoring Browser was considerably more responsive. Most "modern" refactoring tools feel heavy and lugubrious by comparison. The Refactoring Browser was very snappy. Brant and Roberts, the authors of the RB, did this cute demo where they want and replaced every reference to Object in Smalltalk, and transformed Smalltalk into a Thingy-Oriented Programming language. It took under a minute. (This was in VisualWorks 3.1.)
On top of that, you could also write SQL-like queries of the code base in a parser tool, that had full syntactic equivalence to the entire language, with wildcards, and even fully scriptable conditional parse-node matching. You could pop the results of this up in a browser, then rinse, repeat. On top of that, you could use the same parser engine to do wild-card replace! In doing this, you would have gone beyond the "canned" refactorings, so the undo/redo stacks in the Refactoring Browser would no longer be valid, but in Smalltalk, you had the Change Log, which is basically a checkpointed transactional log for all the changes to the image/loaded-codebase. You could even search and filter this log in a GUI tool. As this was entirely local to your install, this was very handy and fast as well. (Until it came time to re-run your change log, and you hadn't saved your image (checkpointed) in like a week. But if you were such a lazy dev, you were getting what you deserved!)
Most of the above has to do with the easy access to the meta-level of the Smalltalk language, combined with the simplicity of the language itself. This makes it 100x easier to write tools, so more tools get written. (1) In particular, very high quality parsing/rewriting tools can get written. Such tools are much lighter weight, and so are more responsive. On the other hand, the high cost of parsing and accessing the meta level in languages like C++, Ruby (parsing only), and Java, just to name a few, has a huge impact on the "landscape" of those tool's programming environments. It's akin to the affect that different kinds of landscape and climate can have on communities of people living there. So the lessons here aren't just for the tool makers. Really, it starts with the language designers, who set what becomes the landscape for the tool makers.
(1) - You can literally start writing a full Smalltalk parser by hand, using top-down LL(1) parsing, have most of the language parsed the same afternoon, and reasonably expect to finish in 1 or 2 days. You can also literally start writing a Smalltalk debugger and have something that lets you browse exception stack frames in 10 minutes.
EDIT: And to head off the predictable "obobjection" -- in VisualWorks 7+ the resulting parser would actually run faster than the equivalent YACC/Lex parser in naive C.
Did my first big project with >200 Unit-Test, like we learned at university, haha.
Well, then requirements changed, and most of them started failing. In the end I spent most of my time fixing the now wrong tests.
On the other hand, stability goes down if I don't do any testing.
At the moment I do automated UI testing. I'm a front-end dev so this seems to catch many things, especially thanks to test-videos and screen-shoots.
I'll try TypeScript in my next production project, simply because I don't know how to write the right amount of good unit-tests. But I know that TypeScript doesn't prevent all bugs, so I guess I'll have a blind spot between the type-checker and the UI tests, but hopefully it won't be too big :\",
In my ideal world, when requirements change, you first change the tests to reflect the new requirements. Then you change code till all the tests pass. Then you're done.
Things don't always work in this manner, but it feels great when they do. UI tests are probably the hardest to write this way.
I think the cost of unit tests when changing code also varies with the quality of the design. If your code manages to some degree to follow the open/closed principle you should be able to just throw out some of your classes. On the other hand that also implies that maybe in those cases the unit tests could have been tossed earlier. However, there always will be things that won't be extended but actually need to be changed and you won't know in advance which is which.
I do however like the idea of unit tests as a scaffold that guide you to build the right thing and then you toss them. Usually I have only very few construction sides in my code base that I am confident enough to burn the scaffold down. It would be an interesting idea to TDD a project and always delete all unit tests after you are done with them and only keep cheaper acceptance tests around. Maybe you would actually ho faster
im a fan of built in tests/traps, like assertions that makes bugs "explode": file x, line 22 did not expect state x to be y. Call Allen. When discovered, make a test that repeats the steps that lead up to the error.
I think most unit testing is a waste because it duplicates what a good type checker would do for you. I'm using TypeScript now, and after a break from using typed languages, it's a huge breath of fresh air.
Much of the code I write doesn't get unit tests at all, because it's simple enough that it won't ever fail. Refactoring major blocks of code is safe, even without unit tests, because the type checker ensures that everything is wired up in a sane manner when you're done. Good design can obviate the need for many unit tests.
When people talk about test code coverage in JavaScript/Ruby/Python, I think the main reason they want close to 100% coverage is that many runtime failures in those languages occur because some line of code somewhere is accessing a type incorrectly. That doesn't happen if you're using static typing.
If you've got some complex logic, making sure it works using unit tests is fine. I still do that with anything I consider non-trivial. But if you've got a really simple function that obviously works, and TypeScript ensures the function will always get the types it's expecting, writing tests to ensure it will keep working forever is just a waste of time, unless it's to verify for your own sake that the "obvious" function does what you think it should. But in that "TDD" case, keeping the test around just makes the code base more brittle, since if you decide you need to change the way the function works you now have two functions to maintain instead of just one.
Really? I saw it mostly in the Ruby community first, followed by Python and JavaScript. Looking at the WikiPedia page, it looks like it came from Extreme Programming and the C3 project [1], which was SmallTalk. And in case you aren't sure, SmallTalk is dynamically typed. [2]
Java has a huge enterprise presence as well, so I'm not too surprised that it's popular in Java circles. Enterprise developers aren't famously top-notch developers in general.
>It's usually the most simple functions that hide the most subtle bugs.
I can't remember the last time a type-checked simple function I wrote hid a subtle bug. But I'm admittedly a few sigmas above average, so my experience is probably not typical.
In studies, TDD is a wash: It neither improves overall programmer throughput nor makes it worse. So use it if it's what makes you happy. I've used it for more complex goals myself, so I understand the draw. I just usually don't need it.
You're funny in the
"Enterprise developers aren't famously top-notch developers in general."
and
"But I'm admittedly a few sigmas above average, so my experience is probably not typical."
I strongly suggest you get off your high horse :)
I also suggest that you read some of the Kent Beck books! Mainly on TDD, he also wrote the JUnit for Java. He considers himself as an average developer who follows good processes... perhaps you're above him as well.
>I strongly suggest you get off your high horse :)
:P
My "a few sigmas above average" comment was to just point out that I'm not typical. It's literally true, though. I really am that good. At least when I'm not writing comments on HN. ;)
Relative to Kent Beck: I've never met him, much less worked with him. But being famous doesn't automatically make you a top 0.1% developer. A great manager and process person? Sure, I'd buy that, based on his books. But based on the relative skill of developers who I've encountered through my career, I'm at least top 1%, and maybe 0.1%. I've met hundreds of developers, and only a very few were in the same league.
I did read Extreme Programming when it first came out. I actually feel that many of the practices in XP, including test-first and pair programming, are actually far more important for average to below-average developers. I think Kent Beck himself said that XP was best for broad-but-shallow problems (things like accounting software with a million simple rules). Broad-but-shallow just doesn't require the same strength of programmer to conquer -- though it does require good process, especially if your team size (and breadth) is large, which is Beck's strength.
JUnit was created by Kent Beck et al to support automated testing with what came to be known as XP and subsequently TDD. This inspired subsequent testing frameworks such as TestNG and implementations for other languages like nUnit.
The obsession with information theory here seems like a classic nail-hammer thing. The number of bits my tests convey is totally useless to think about and certainly not worth spending pages on. All I want from my tests from a code base I maintain for thousands of patches is a tiny fraction of a bit: did my latest change break an important behavior or invariant encoded in a unit test? If I only screw up once in every 100 patches, then formally, my unit tests are doing all that work to emit 0.01 bits of information (-log(99/100)), which is formally a totally irrelevant thing to know about my unit testing framework. ('Hey Joe, what have you been up to?' 'Fixing my unit testing framework - I'm up to 0.03 bits per patch!' 'I see.')
Reading over the comments, a lot of people seem to be concerned that unit tests compromise their ability to change code quickly as requirements change - they find themselves spending too much time updating tests instead of doing real implementation work.
In the experimental stages of a project, I'd buy that. But once a project has matured to the point where it's working and the architecture is broadly in place, requirements changes are usually not so fundamental that there is no resemblance between the old and new requirements. If you're finding yourself having to rewrite large swathes of unit tests when the requirements change, you need to ask yourself if the real problem is that your code is poorly factored. If you're breaking down your code into simple, independent, cleanly composable pieces, you'll find that changing requirements poses much less of a testing burden.
Unit tests enforce good design and naturally provide decent documentation on how components fit together. Unit tests also allow you to spend a bit more time on the details of each component and find issues you'd otherwise find in production. If nothing else at least have some integration tests so that you can confidently refactor knowing that you're not breaking existing functionality somewhere.
One key thing the author started to allude to: the value of functions - pure functions, vs a cyclic graph of mutable objects with temporal coupling of state changes and spaghetti inheritance out the wazoo.
OK, he didn't say that exactly, but he really did start heading that way.
The "static types will save us crowd" is drowning out a lot of the rest of the discussions that need to happen.
* Somebody mentioned Eiffel in other comments. Eiffel's design-by-contract assertions about invariants are vastly preferable to Java's magic beans that eventually (maybe) reach a valid useful state. I guess unit tests sort of compensate for this, but not really.
* Mutable state needs to be pushed to the peripheries of our apps, not plastered everywhere.
* Inheritance might not be such a good idea, due to reasoning about which code runs when you call a method / send a message. Do like Go-lang, and have interfaces for polymorphism, but skip inheritance - the use of an interface is a flag for later binding. Likewise, higher order functions and closures help you reuse code without resorting to spaghetti inheritance.
* We need languages that make programming with types easy, using type inference where it is clear to do so (e.g. - local identifier initialization, but perhaps not for multi-line function headers?). BUT, we still need to allow for dynamic runtime types, perhaps with a few minimal flags on identifiers/modules, rather than forcing people into monstrous reflection frameworks and XML situps.
FWIW, I agree (with the paper) that feature/integration tests are very good, but unit tests are often a waste.
69 comments
[ 2.7 ms ] story [ 128 ms ] threadEveryone please stop testing your code.
...
...
...
...
Ok folks - everyone who's still testing can keep their jobs (or take the jobs of those who stopped).
Snap summaries make for reflexive reactions. That's the shallow kind of discussion we're hoping to avoid here. What we want is reflection, which is slower, takes more energy, and leads to more considered exchanges.
More on the reflexive/reflective distinction: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
There's a lot of long rubbish on the Internet. Being able to quickly distinguish it from good stuff is essential to using your time sensibly.
Frankly if you aren't willing to read the content you simply shouldn't comment on it until you have.
Frankly, if you aren't capable of summarizing the content, any other comment you'd make is probably rubbish.
The problem is tl;dr culture, which actively encourages people not to read the story, and, worse, to discuss the story without having read it.
This is so easy: you can almost entirely avoid the issue simply by avoiding the term "tldr".
See also: https://henrikwarne.com/2014/09/04/a-response-to-why-most-un...
This seems.. entirely unreasonable to me. Just because some area of code is not touched frequently doesn't mean that we should throw out tests pertaining to that section of code. Can someone justify this to me? This seems like a _terrible_ mistake.
Regression tests are mentioned later and are not what the author is addressing here.
Still make sense?
I would really like some system that can tell me how many times a line of production code has executed in the last year. For an ecommerce web app, I can't imagine how a line of code that hasn't run in the last two years would still need to be there.
"""
Gregory (Scotland Yard detective): Is there any other point to which you would wish to draw my attention?
Sherlock Holmes: To the curious incident of the dog in the night-time.
Gregory: The dog did nothing in the night-time.
Holmes: That was the curious incident.
"""
Provided you're following good unit testing practice, I can't imagine an organization whose best investment is removing excess unit tests.
Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring.
Tests take time to run, which slows the development loop.
Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation.
When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily.
"Has been passing for a long time" isn't necessarily a good heuristic for identifying these situations. I'm just saying there are lots of costs to having a passing test in the suite.
Agreed, but...
> Sometimes they rely on interfaces other than the one being tested, and that code needs to be updated when those interfaces updated. Generally speaking, an interface will have 2x the consumers in a well tested codebase, which means 2x the refactoring.
Only when refactoring the public interface, and only when tooling isn't capable of automating the refactoring. In other words, only when you're doing something complex enough that there's a high chance you'll break things and cause tests to fail!
Hopefully one will catch said test failures before checking in - note that this will skew any CI metrics. Just because they didn't fail on your CI infrastructure within the past year, doesn't mean they didn't fail within the past year.
> Tests take time to run, which slows the development loop.
This is, perhaps, a good reason to make those tests run e.g. only for your overnight integration tests. But removal? Nervously auditing and reviewing changes (both your own, and other's in code reviews) for possible breaks because of reduced confidence that your unit testing will catch things also slows down the development loop. As does the debugging sessions when that isn't enough.
> Test can be used as documentation. The more tests in a specification, the less useful that spec is as documentation.
This is, perhaps, a good reason to relegate less informative tests to another directory. Perhaps the same directory that houses regression tests that your fuzzer found.
> When refactoring, the number of tests increases the amount of behavior that needs to be pulled along. If the test is no longer important to the business, that slows down refactoring unnecessarily.
The purist might cry that refactoring doesn't change (external) behavior. But I'll note that it's just as easy to delete unit tests when you're changing behavior, as it is to delete them ahead of time. Even easier! Bonus: Deferring their deletion until then makes the fact that you're actually changing behavior a lot more obvious and explicit.
Ahh, a fibbed a little: It might be harder to get those deletions approved by your code reviewer if it's not obvious that changing behavior is OK. In my work environments, this is a feature.
I can at least confirm with certainty that it sometimes is a terrible mistake.
I port things. One approach I sometimes use is to stub in noop implementations of platform specific things. I generally use #pragmas or the like to generate a warning to come back to (which I can always later crank up to an error to ensure I've gone back and implemented everything.) It's common that the original authors were also "kind enough" to leave some noop implementations of their own - although usually not with the helpful #pragmas.
End result: Thousands of failing unit tests, most of which were passing for more than a year. Some are innocuous as minor string formatting changes caused by minor libc differences - but even that can be concerning if you want consistent, reliable results that don't vary by platform and threaten to cause platform specific edge cases.
I've never seen a codebase where even 10% of the failures are that innocuous.
Unit tests only make sense for components of a system which are BOTH critical and stable. I think that integration tests are often way more useful for most projects.
In some environments, refactorings to the code will also refactor the tests. Doing test first is only a win in the situation where you don't have to duplicate work, and only if management wouldn't otherwise give you the time to write tests after the fact. Unfortunately, the former is less common, and the latter is more common.
1. Keep regression tests around for up to a year - but most of those will be system-level tests rather than unit tests. [System-level since context of how software was being used is important.]
2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
3. Except for No 2, if X has business value and you can use a system or unit test use a system test for X. [Context again.]
4. Design a test with more care than you design the code. [They need to do something meaningful rather than just be there as a metric.]
5. Turn most unit tests into assertions. [Assertions describe properties you code should always have before, during, or after executions. Just use assertions for such checks that instead of hiding them in tests.]
6.Throw away tests that haven’t failed in a year. [Controversial claim. Says they tell you nothing. I believe the author thinks the design and assertions should make the software right from the beginning with the tests telling you where you're screwing up.]
7. Testing can't replace good development: a high test failure rare suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth.
8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers performance. Rewarding coverage or some other meaningless metrics can lead to rapid architectural decay.
9. Be humble about what tests can achieve. Tests don't improve quality: developers do.
There's a lot of good points in there, though. Talks about combinatorial explosion where tests often don't even measure correctness. Talks about how maintenance burden goes up over just good system testing and assertions with minimal unit testing. Meantions Toyota Production System showed you keep human in the loop and let them do any analysis that requires brains while automating just the mundane stuff. Mentions how hardware engineers duing Design-for-Testing philosophy embed little probes in most of their blocks to catch violation of correctness conditions during testing. Kind of black box plus deep, white box. Says that systems could similarly be wired so the system tests would set off alarms when they should. Just a few gems I saw.
I agree a shorter version of this essay would be beneficial.
When doing "dynamic" programming (Java + Spring + Maven, Python, anything JavaScript), I feel compelled to write more unit tests. Because the sand shifts under my feet and my crap breaks in surprising ways.
I despise most modern development, where entropy continues to creep in and nothing stays done.
Maybe more assertions would help mitigate my feelings of hopelessness and despair.
https://www.eiffel.com/values/design-by-contract/introductio...
It has been implemented in Java with the stuff inside JavaDocs. I know JML/Krakatoa integrated it with provers. Simplest is using it for interface checks, though, of expectations of what component is doing vs what it's actually doing. The DbC style lets you document your assumptions, have formal properties to derive tests from, work right there with the code itself, and even automatically generate tests in some tools. That last one is huge benefit as you know the tests will always be relevant to the module and don't have to maintain them. You can do your own tests at the system-level on stuff the module-level contracts might not capture.
I'm not sure I can help you for most of the others. Remember, though, you can implement the DbC assertions in code that's in the modules themselves called before and after they run. These can be set to turn off with a flag during production compiles. You can also write generators that pull them out of comments in your source files then produce a version of your program with the checks in them as tests. I can't find my original articles post-PC crash but the one below builds in Ruby a generator for C language DbC:
http://www.onlamp.com/pub/a/onlamp/2004/10/28/design_by_cont...
EDIT: Two for Python on StackOverflow that are kludgey but seem to help.
https://stackoverflow.com/questions/8563464/using-design-by-...
As an aside, I've worked on projects where coverage wasn't used and the rule was to "test the important bits" and I've also been on projects where coverage had to be >= ~98%. I wonder if a middle-ground approach would be effective.
I hate it when people write tests to cover every single function (to cover every line of code) but they pay almost no attention to the range of inputs/arguments passed to those functions.
I think that most bugs in production can be traced back to "An unexpected argument which was passed to a complicated function/method" - According to this, you should focus your testing efforts on methods/functions that have complicated logic and you should test them with a range of unusual arguments/inputs.
Writing tests for EVERY function/method using only obvious (common use-case) arguments is a waste of time.
But what happens is the unit tests become filled completely useless logic. Like a test with only mock objects that just tests if a method can be called. Useless but good for padding those code coverage numbers. And ironically it greatly decreases code quality. Because when there's these useless unit tests covering everything, it makes refactoring very difficult. And so no one refactors to improve the code because its too much work.
We had this solved in Smalltalk in the early 2000's. Why has the rest of the programming industry screwed the pooch on this, since?
Our standard refactoring tool, the original Refactoring Browser (RB), covered Unit Tests just the same as any other code. If you fired off a "canned" refactoring in it, it was guaranteed to be correct, you could multiply undo/redo it, and it covered everything currently loaded in the image, which typically included Unit Tests.
I notice in Visual Studio and also with CLion and some other "refactoring" tools for C++ I've played around with, that some "refactorings" amount to: "We'll generate/move code for you, but you're on your own with regards to correctness." This reduces the usability and productivity of the tool considerably.
In addition, the Refactoring Browser was considerably more responsive. Most "modern" refactoring tools feel heavy and lugubrious by comparison. The Refactoring Browser was very snappy. Brant and Roberts, the authors of the RB, did this cute demo where they want and replaced every reference to Object in Smalltalk, and transformed Smalltalk into a Thingy-Oriented Programming language. It took under a minute. (This was in VisualWorks 3.1.)
On top of that, you could also write SQL-like queries of the code base in a parser tool, that had full syntactic equivalence to the entire language, with wildcards, and even fully scriptable conditional parse-node matching. You could pop the results of this up in a browser, then rinse, repeat. On top of that, you could use the same parser engine to do wild-card replace! In doing this, you would have gone beyond the "canned" refactorings, so the undo/redo stacks in the Refactoring Browser would no longer be valid, but in Smalltalk, you had the Change Log, which is basically a checkpointed transactional log for all the changes to the image/loaded-codebase. You could even search and filter this log in a GUI tool. As this was entirely local to your install, this was very handy and fast as well. (Until it came time to re-run your change log, and you hadn't saved your image (checkpointed) in like a week. But if you were such a lazy dev, you were getting what you deserved!)
Most of the above has to do with the easy access to the meta-level of the Smalltalk language, combined with the simplicity of the language itself. This makes it 100x easier to write tools, so more tools get written. (1) In particular, very high quality parsing/rewriting tools can get written. Such tools are much lighter weight, and so are more responsive. On the other hand, the high cost of parsing and accessing the meta level in languages like C++, Ruby (parsing only), and Java, just to name a few, has a huge impact on the "landscape" of those tool's programming environments. It's akin to the affect that different kinds of landscape and climate can have on communities of people living there. So the lessons here aren't just for the tool makers. Really, it starts with the language designers, who set what becomes the landscape for the tool makers.
(1) - You can literally start writing a full Smalltalk parser by hand, using top-down LL(1) parsing, have most of the language parsed the same afternoon, and reasonably expect to finish in 1 or 2 days. You can also literally start writing a Smalltalk debugger and have something that lets you browse exception stack frames in 10 minutes. EDIT: And to head off the predictable "obobjection" -- in VisualWorks 7+ the resulting parser would actually run faster than the equivalent YACC/Lex parser in naive C.
Well, then requirements changed, and most of them started failing. In the end I spent most of my time fixing the now wrong tests.
On the other hand, stability goes down if I don't do any testing.
At the moment I do automated UI testing. I'm a front-end dev so this seems to catch many things, especially thanks to test-videos and screen-shoots.
I'll try TypeScript in my next production project, simply because I don't know how to write the right amount of good unit-tests. But I know that TypeScript doesn't prevent all bugs, so I guess I'll have a blind spot between the type-checker and the UI tests, but hopefully it won't be too big :\",
Things don't always work in this manner, but it feels great when they do. UI tests are probably the hardest to write this way.
Much of the code I write doesn't get unit tests at all, because it's simple enough that it won't ever fail. Refactoring major blocks of code is safe, even without unit tests, because the type checker ensures that everything is wired up in a sane manner when you're done. Good design can obviate the need for many unit tests.
When people talk about test code coverage in JavaScript/Ruby/Python, I think the main reason they want close to 100% coverage is that many runtime failures in those languages occur because some line of code somewhere is accessing a type incorrectly. That doesn't happen if you're using static typing.
If you've got some complex logic, making sure it works using unit tests is fine. I still do that with anything I consider non-trivial. But if you've got a really simple function that obviously works, and TypeScript ensures the function will always get the types it's expecting, writing tests to ensure it will keep working forever is just a waste of time, unless it's to verify for your own sake that the "obvious" function does what you think it should. But in that "TDD" case, keeping the test around just makes the code base more brittle, since if you decide you need to change the way the function works you now have two functions to maintain instead of just one.
It's usually the most simple functions that hide the most subtle bugs.
Java has a huge enterprise presence as well, so I'm not too surprised that it's popular in Java circles. Enterprise developers aren't famously top-notch developers in general.
>It's usually the most simple functions that hide the most subtle bugs.
I can't remember the last time a type-checked simple function I wrote hid a subtle bug. But I'm admittedly a few sigmas above average, so my experience is probably not typical.
In studies, TDD is a wash: It neither improves overall programmer throughput nor makes it worse. So use it if it's what makes you happy. I've used it for more complex goals myself, so I understand the draw. I just usually don't need it.
[1] https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
[2] https://en.wikipedia.org/wiki/Smalltalk-80
and
"But I'm admittedly a few sigmas above average, so my experience is probably not typical."
I strongly suggest you get off your high horse :)
I also suggest that you read some of the Kent Beck books! Mainly on TDD, he also wrote the JUnit for Java. He considers himself as an average developer who follows good processes... perhaps you're above him as well.
:P
My "a few sigmas above average" comment was to just point out that I'm not typical. It's literally true, though. I really am that good. At least when I'm not writing comments on HN. ;)
Relative to Kent Beck: I've never met him, much less worked with him. But being famous doesn't automatically make you a top 0.1% developer. A great manager and process person? Sure, I'd buy that, based on his books. But based on the relative skill of developers who I've encountered through my career, I'm at least top 1%, and maybe 0.1%. I've met hundreds of developers, and only a very few were in the same league.
I did read Extreme Programming when it first came out. I actually feel that many of the practices in XP, including test-first and pair programming, are actually far more important for average to below-average developers. I think Kent Beck himself said that XP was best for broad-but-shallow problems (things like accounting software with a million simple rules). Broad-but-shallow just doesn't require the same strength of programmer to conquer -- though it does require good process, especially if your team size (and breadth) is large, which is Beck's strength.
There are programming languages, frameworks and tools which would be unlikely to exist without "advances" such as TDD.
In the experimental stages of a project, I'd buy that. But once a project has matured to the point where it's working and the architecture is broadly in place, requirements changes are usually not so fundamental that there is no resemblance between the old and new requirements. If you're finding yourself having to rewrite large swathes of unit tests when the requirements change, you need to ask yourself if the real problem is that your code is poorly factored. If you're breaking down your code into simple, independent, cleanly composable pieces, you'll find that changing requirements poses much less of a testing burden.
OK, he didn't say that exactly, but he really did start heading that way.
The "static types will save us crowd" is drowning out a lot of the rest of the discussions that need to happen.
* Somebody mentioned Eiffel in other comments. Eiffel's design-by-contract assertions about invariants are vastly preferable to Java's magic beans that eventually (maybe) reach a valid useful state. I guess unit tests sort of compensate for this, but not really.
* Mutable state needs to be pushed to the peripheries of our apps, not plastered everywhere.
* Inheritance might not be such a good idea, due to reasoning about which code runs when you call a method / send a message. Do like Go-lang, and have interfaces for polymorphism, but skip inheritance - the use of an interface is a flag for later binding. Likewise, higher order functions and closures help you reuse code without resorting to spaghetti inheritance.
* We need languages that make programming with types easy, using type inference where it is clear to do so (e.g. - local identifier initialization, but perhaps not for multi-line function headers?). BUT, we still need to allow for dynamic runtime types, perhaps with a few minimal flags on identifiers/modules, rather than forcing people into monstrous reflection frameworks and XML situps.
FWIW, I agree (with the paper) that feature/integration tests are very good, but unit tests are often a waste.