There's a human tendency to overemphasize things you can quantify. So we try to figure out how to test every code path rather than what we should do: try to figure out which inputs we should test against.
I have 100% code coverage on a couple of projects. It has two benefits:
Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected.
New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the standard "If your contribution doesn't have 100% coverage it won't be accepted". I don't think this is a bad default position.
The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.
So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually test any known edge cases.
Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.
A good example would be in dynamically typed languages where you can hit a code path but forget to validate a certain input type which could cause a bug.
Or one I've been personally hating lately, a nil pointer dereference in Golang from a well tested library that didn't do proper error propagation.
> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.
This is true.
But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)?
I think that a greater level of confidence in our code can be achieved by a combination of
- Judicious choices of unit and integration tests
- Static and dynamic analysis (if the language supports it), and
- Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases.
As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example.
If 100% coverage cannot be achieved, we need to ask ourselves:
- Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things?
- Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it?
- If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect.
- If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed.
> If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it.
More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x."
Exactly. It's all about logic. They other types of bugs are caught by the IDE, the compiler or the first run anyway.
Let someone else look over your code, they might spot logic bugs that you oversee. If you write tests yourself, you will often oversee such logical bugs.
I have done (semi-)automatic reasoning and code verification with 100% code coverage of OS drivers. All the mathematical reasoning doesn't spot logical errors. You need more than one person to look over it.
It's slightly more meaningful in the right hands, but still meaningless in the wrong ones. You can get 100% line coverage and never have a single assert.
100% code coverage and even TDD'd doesn't and shouldn't mean 100% unit tested. Glue code and declaration doesn't need a unit test. Some functional tests should provide all the coverage needed to give you confidence to refractor that code in the future.
Edit: while I'm a huge TDD advocate, I'm not a big advocate of measuring code coverage. That should only be necessary if you are trying to get a code base under coverage that wasn't TDD'd. Even then I'd rather add the coverage as I'm touching uncovered code. If it works and I'm not touching it, it doesn't need tests.
I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this.
Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went untested. A single execution isn't even enough for me: I need a whole lot of testing on most of the code to be comfortable.
When I write Scala at work instead, I test algorithms, but a big percentage of my code is untested, and it all feels fine, because while not every piece of code that compiles works, the kind of bugs that I worry about are far smaller, especially if my code is type heavy, instead of building Map[String,Map[String,Int]] or anything like that. 100% code coverage in Scala rarely feels as valuable as in Ruby.
Also different styles make the value of having tests as a way to try to force good factoring changes by language and paradigm. Most functional Scala doesn't really need redesigning to make it easy to test: Functions without side effects are easy, and are easier to refactor. A deep Ruby inheritance tree with some unnecessary monkey patching just demands testing in comparison, and writing the tests themselves forces better design.
The author's code is Java, and there 95% of the reason for testing that isn't purely based on business requirements comes from runtime dependency injection systems that want you to put mutability everywhere. Those are reasons why 100% code coverage can still sell in a Java shop (I sure worked in some that used too many of the frameworks popular in the 00s), but in practice, there's many cases where the cost of the test is higher than the possible reward.
So if you ask me, whether 100% code coverage is a good idea or not depends a whole lot on your other tooling, and I think we should be moving towards situations where we want to write fewer tests.
IMO ruby is the worst because its so easy, and often times encouraged, to write layers of useless tests that ultimately give only a false sense of security. I recently re-did the Michael Hartl tutorial to catch up with Rails5 and it gives examples where you test proper template rendering. I love that book but I'd rather shoot myself in the foot then spend dev time writing tests to check if the right html title attribute was rendered for the page...
Sometimes the right test is just occasional manual testing. And sometimes it's even just the fact that no users have complained about something obvious not being completely wrong.
Curious as to why you're blaming a language (ruby) for a bad practice (writing layers of useless tests). Excuse my pedantry, but it wouldn't it be more accurate to blame a culture that you feel has grown up around that language?
In Scala, "does it compile/typecheck" will generally catch typos in code that isn't frequently executed, and raise warnings for unhandled cases. For Ruby, you pretty much need to execute the code to have any inkling if it goes kaboom -- the language (and how people write it) have so much magic that short of executing things, the computer can't tell you anything useful.
I don't think I could write low-bug count code faster in statically typed scala than I could in unit tested ruby though. I mean I am well aware of what you're telling me, and it's obvious to me that having the compiler automatically check certain properties is a win. And yet, when push comes to shove, to get something done I'm more likely to reach for ruby.
It's something I've never come up with a good explanation for. Does static typing stunt prototyping and exploration? Do unit tests capture high level goals better?
I guess I don't agree with the assertion that ruby tests are useless. Because you test higher level things than you do with scala types.
But you're kind of implying that I feel that way because either I don't use static typing much, or I haven't learned it well, aren't you?
I've used static typing much more than dynamic. I'll admit confusion on things like ocaml polymorphic variants and haskell monads, nevertheless I wouldn't say I find static typing hard as a rule. But surely the point of static type checking is to constrict what you can do for safety and performance reasons. And surely the cost of that is you have less freedom - even when in the exploration phase
"you're kind of implying that I feel that way because either I don't use static typing much, or I haven't learned it well, aren't you?"
No. I didn't mean to imply or assert anything of the sort. Apologies if I inadvertently gave offense. You seem to be reading meanings into my reply that aren't there.
I don't know you from Adam. You asked a question in your post. I answered as best as I could.
That said, your latest statement
"I'll admit confusion on things like ocaml polymorphic variants and haskell monads,"
does seems to imply that you don't have much real world experience with static type systems in production (nothing wrong with that) since neither is an arcane concept or particularly difficult to understand.
If you haven't worked extensively with Haskell/Ocaml/SML etc, and you are extrapolating properties of 'static type systems' from those of Java or C++ then your idea of such type systems 'stunt prototyping and exploration' might make sense.
The rest of your comments are extrapolations from misunderstandings - not born of practical experience. My answer was based on (strictly) personal experience. Which is why I said "in my experience". I gladly concede that YMMV.
again, I was just answering your question in your original comment. I didn't mean to "imply" anything and used "probably" to mark my uncertainty about your real world experience with Ocaml/Haskell etc style static type systems.
I have extensive experience with both dynamically typed languages (mostly Python, Lua and Scheme) and statically typed languages (mostly Haskell and SML, besides Java). I answered out of my experience.
I'm not offended. I am happy to talk to open minded static typing advocates. I myself am not really sure where I sit.
You're right, I haven't used ocaml or haskell in production. I did use F# though, which seems to be in the same ballpark as those languages in terms of having algebraic data types and inference and all the rest. I suppose the fact that I still don't really grok polymorphic variants after reading the real world ocaml a few times may say something about my ability, motivation, or at the very least how my brain is wired.
Fundamentally though, a type system is a restriction meant to help the programmer. This restriction must inevitably put you down a certain road when you explore stuff, right?
It changes my approach to a problem, in that, with dynamic typing, I tend to think of problems as manipulating data, while with static typing I am modeling a domain, deseriazing data to it, and serializing the result. The end result usually ends up being a bit more verbose, but at the same time does a better job of holding my hand when the input format changes in the future. It's definitely a trade off though.
If you can achieve >90% testing, efficiently and properly. It's worth it. Testing for the correct title is worth it. Especially if the title has something appended to it at render time (i.e. hello | website_name). Someone can over ride this on a page and having a test to catch that is important. A well written test would check for page_name | website_name across all pages. Pages are finite. Such tests are useful. Hartl uses basic testing framework as an introduction but there are better ones available.
I think I'd rather focus on documenting the information flow. Of having the tools to track down where things start to go wrong when there's a problem and I ask things to run with more verbosity.
Initial "complete coverage" should probably start from mockups that test an entire API. The complete part should be that, in some way, the tests cover expected successes AND failures (successfully return failure) of every part of the API, but there's no need to test things individually if they've already been tested by other test cases.
Invariably reality will come up with more cases and someone will notice an area that wasn't quite fully tested. That's where a bug exists, but the golden test cases probably wouldn't have located it anyway. It'll take thousands or millions of users to hit that combination and notice it. Then you get to add another test case while you're fixing the problem.
I've seen projects where management had rules like "you must have 70% code coverage before you check in". Which is crazy, for a lot of reasons.
But the developer response in a couple cases was to puff the code up with layers of fluff that just added levels of abstraction that just passed stuff down to the next layer, unchanged, with a bunch of parameter checking at each new level. This had the effect of adding a bunch of code with no chance of failure, artificially increasing the amount of code covered by the tests (which, by the way, were bullshit).
I got to rip all that junk out. It ran faster, was easier to understand and maintain, and I made sure I never, ever worked with the people who wrote that stuff.
I wish people cared more about the craft of an amazing plugin architecture or an advanced integration between a machine learning system and a UI, but no, more and more of our collective development departments care more about TDD and making sure things look perfect. Don't worry about the fact that there are no integration tests and we keep breaking larger systems, and while there might be 100% code coverage, no developer actually understands the overall system.
Yes, unit tests have their place, but you have to understand why they are needed.
To me, it's secondary and has tones of bike shedding in it. Writing tests is easy, mentally. Getting a good, simple, YAGNI/DRY architecture is more challenging and requires several iterations, something that is resisted if you have to rewrite all your unit tests. Let me put it this way, if you write an architecture that you later don't like, you would be more hesitant to scrap it and start over because all the additional work of the unit tests (especially if you're on a deadline). That's bad. To write a good architecture, a developer must be willing to realize he could have done it better and be willing to tear it down instead of build around the bad architecture. That's how you write 20 year code, which should be almost everyone's goal.
I think a bigger epidemic is we're putting too much emphasis on "do this" and "do that" and "if you don't do this then you're a terrible programmer". While that sometimes may be true, much more importantly is to have competent, properly trained professionals, who can reason and think critically about what they're doing, and who have a few years of experience doing this under their belt. Just like other skilled trades, there's a certain kind of knowledge that you can't just explain or distill into a set of rules, you have to just know it. And I see that in the first example in this article, where the junior programmer is writing terrible tests because he just doesn't know why they're bad tests (yet).
It seems to me that management is taught that a dependence on expensive experts, is a problem to be optimized. They want to manage the development process in a way that allows them to easily swap out one developer or team for another, or to ramp up production simply by increasing the number of developers assigned to a task.
It is almost as if they see the success that we have had in dev/ops with the "cattle, not pets" philosophy, and want to apply that in their own field. Making the subject behave consistently and predictably, whether it is a machine or a human professional, would be a prerequisite for that.
I'd say they are "succeeding" in the sense that they continue to make money. Whether they're truly successful in the sense of providing proper value to their customers to justify that money, and treating their programmers right, is another story, and I've seen enough cases where they don't do either of these, while the executives make plenty of money to live comfortably for years by exploiting both customers and their own programmers.
On the other hand, every discipline that I can think of has its own set of best practices. Why should software development be any different? I know a lot of people are susceptible to the mindless repeating, but that's not a fault of best practices.
I'm not against the idea per se, just highly skeptical when I hear it used, because more often then not, I've seen it used to promote mindless repetition of previously used patterns.
Naming the test just "initialise" is not very useful as it doesn't assert what you expect the method under test to do. Given that the purpose of the initialise function is to populate a watchlists collection variable from the parameter, i'd name the test something like "initialise_daoRecordCountIs9_watchlistCountIs9". The pattern I generally use is <method_name>_<assertion_under_test>_<expected_result>.
Then, my test would be the following:
* Set up / mock the dao parameter to have 9 rows
* Create an instance of the class under test and push in the dao parameter
* Verify / Assert that the class under test now has 9 items in the watchlists variable - I'm assuming there is a public method to access that.
I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases.
Just because you ran a function or ran a line doesn't mean it will work for the range of inputs you are allowing. If your function that you are running coverage on calls into the OS or a dependency, you also have to be ready for whatever that might return.
Therefore you can't tell if your code is right just by having run it. Worse, you might be lulled into a false sense of security by saying it works because that line is "covered by testing".
The real answer is to be smart, pick the right kind of testing at the right level to get the most bang for your buck. Unit test your complex logic. Stress test your locking, threading, perf, and io. Integration test your services.
There's use in looking at the testing for industrial strength code e.g. compilers, DirectX, OpenGL conformance (god help you). With that background, I was confused by TDD to put it politely.
This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range.
When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself.
As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems
> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem.
One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way.
Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.
This depends on how you think of things. Some languages (Eiffel?) have pre- and post-conditions, which do some of this job. Much of the boilerplate testing done for dynamic languages is done by the compiler for static languages. The borrow checker in Rust is doing the work that a test suite and/or static analysis tool would be doing for C/C++
Rust has simple unit testing built into the language. And in Ada/Spark, tests can be a first class verification tool, alongside formal verification.
We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger tools should be able to hone in on areas where the result diverges from expectations.
And another example of something I'd want checked by the language is exception throwing / handling. It's another one of those places where code coverage won't help you unless you already know what you're looking for. Languages are getting better about it, but in general, handling errors is hard.
Force.com doesn't allow you to deploy code to production unless at least 75% of the statements are executed[0]. I wish my employer had a similar requirement.
> IMO, a unit that does not pass a spec/test should cause a compile time error.
We can achieve this in dependently-typed languages like Idris. First we define a datatype 'Equal x y', which will represent the proposition that expression 'x' is equal to expression 'y':
data Equal x y where
Refl : (x : _) -> Equal x x
There are two things to note:
- There is only one way to construct a value of type `Equal x y`, which is to use the constructor we've called `Refl`.
- `Refl` only takes one argument, `x` (of arbitrary type `_`), and it only constructs values of type `Equal x x`. This is called "reflexivity", thus the name `Refl`.
Hence if we use the type `Expr x y` anywhere in our program, there is only one possible value we can provide that might typecheck (`Refl x`), and that will only typecheck if `Expr x x` unifies with `Expr x y`, which will only happen if `x` and `y` are the same thing; i.e. if they are equal. Thus the name `Equal`.
This `Equal` type comes in the standard library of all dependently typed languages, and is widely used. To use it for testing we just need to write a test, e.g. `myTest`, which returns some value indicating pass/fail, e.g. a `Boolean`. Then we can add the following to our program:
This will only type-check if `Equal myTest myTest` (the type of `Refl myTest`) unifies with `Equal myTest True`, which will only be the case if `myTest` evaluates to `True`.
It's been very effective at improving the overall quality of the code. And because of CTFE (Compile Time Function Execution) and static assert, many code correctness tests can even be run at compile time.
I think there's an argument to be made that if the desire for testing is a main driver for your architecture then your tests are too granular, and you aren't testing any internal integration points. In my experience that means that your tests are so tied to your current architecture that you can't even refactor without having to rewrite tests.
mocking / interaction / expect breaks encapsulation to perform "testing".
Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually:
1) fragile to implementation
2) opaque as to intent
Whereas input/output integration points are more reliable, transparent, and less fragile to implementation changes if the interface is maintained.
However, if you must do mock-level interaction testing, Spock has made it almost palatable in Javaland.
This is one area where functional fans get to make the imperative folks eat their lunch.
I think that depends on the language/environment... for example, it's usually MUCH easier to get high coverage, and to do just enough mocking testing against Node-style modules in JS than logic in a typical C# or Java project.
Usually because interfaces need to be clearly/well defined and overridden... In JS there are tools (rewire, proxyquire, etc) that can be used with tests in order to easily inject/replace dependencies without having to write code to support DI nearly as much. In fact, I'd say that there's usually no reason not to be very close to 100% coverage in JS projects.
> If you have many methods in a class with only one basis path, what purpose does the class serve.
It ties together related algorithms. You want "addDays(n)", "addHours(n)", "addMinutes(n)" and so on to be in the same class, even if they're one-liners.
Clearly there are good examples either way. I'm more interested in the idea that this strong mismatch between the complexity of the code and the complexity of its tests is, in itself, a code smell that may point to an issue that is nothing to do with TDD.
I note that the examples you give have trivial test cases - presuming you don't care about overflow, and if you do, then those methods now have basis paths, whether explicit or implicit depends on the language.
"Pex uses a constraint solver to produce new test inputs which exercise different program behavior. "
So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/
Still a pity it's only included in Enterprise edition. We need a strong foundation of accessible tools for automated test input generation improving on the current state of the art fuzzers. See e.g. danluu's vision of "Combining AFL and QuickCheck for directed fuzzing" [https://danluu.com/testing/].
This is what I love about the design and operational philosophy of the Erlang/OTP evironment: the trickier class of harder-to-test/reproduce transient/timing pre-defensive-coding issues can sometimes be resolved automagically under supervision trees restarting said Erlang process. Log the stacktrace as a failure to maybe be fixed and move on. That's in addition to a robust soft-realtime, SMP VM which allows remote, live debugging.
As a newbie to Erlang, how does Erlang handle this case better for a service than a Python process which logs and restarts on failure? Also, this runtime testing strategy doesn't handle cases where you have other systems depending on that Erlang code path being successful, does it?.
Code coverage should perhaps not be counted in lines, but in the number of type permutations covered. Then your challenge is determining all the types that _can_ be passed as inputs.
Many embedded safety and medical applications prove correctness for all inputs and their combinations. Somme also verify error behaviour. (Out of range.) Granted, this is a relatively small input space, typically a few sensors.
One of them would be a fuzz testing.
You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes.
The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches.
It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast).
Fuzz testing is useful at testing whole systems in vivo.
Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code.
Depending on your means and needs, you want to try both.
Parent was talking about full coverage. While fuzzers are great tools, able to generate test inputs for easily coverable parts of the code, formally correct software has to be designed for provability.
In practice, testing _all_ possible inputs is usually not a feasible goal, and often downright impossible (since it would mean solving the halting problem).
While there are some languages and problem domains where formal verification methods are a possibility, for most of software development the best you can do is reducing solution space.
A sane type system would be a start for example.
And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well.
TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs.
But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.
Polyspace [0] allows full state space checking. Usually, this is feasible because the kind of modules you test are heavily constrained 12 bit (or fewer) fixed point values (sensor output/ actor input) and contain no algebraic loops. With floats and/or large integers, this quickly becomes unwieldy (verification then takes weeks).
I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible.
Brute force.
Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results.
Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field.
Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.
How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.
… this reply does nothing to answer the parent's question. It's obvious that you can run it on all possible values, but that doesn't answer the question of the return values. (unless you only want to verify "doesn't crash" or something similarly basic)
I believe the question was edited after I wrote my answer. This, or I just misread it - unfortunately HN doesn't let us know. Anyway, I'm providing another answer to the question as it is now.
Your comparison source could be running on a different platform, or be slower, or ... and thus not be a drop-in replacement. (E.g. in the example of a broken floating-point unit, compare the results with a software emulation)
Hmm, true, it may have been a different platform/slower, thanks. It can't have been hardware/software, as the GP said they were testing a library (which implies software).
It's a good idea in principle, though your third and fourth properties are not true of floating point addition.
When you have one available, testing against the results against a reference implementation is a little more direct. Though, given how easy those property tests are, you might as well do both.
How is the algorithm supposed to know what the right thing to return is
Some times I have, for example, matlab code that I 'know' is right and want to verify that my C/Python/Julia... code does the same thing. So then I just call the matlab code, call the new code and check if they're the same.
Or I have a slow exact method and want to see if my faster approximation always gives an answer 'close enough' to the correct answer.
With floating-point math libraries especially, it's frequently the case that it's relatively easy to compute the desired result accurately by using higher precision for internal computations; e.g. you might compare against the same function evaluated in double and then rounded to float.
This sounds pretty sketchy at first, but it turns out that almost all of the difficulty (and hence almost all of the bugs) are in getting the last few bits right, and the last few bits of a much-higher precision result don't matter when you're testing a lower-precision implementation, so combined with some basic sanity checks this is actually a very reasonable testing strategy (and widely used in industry).
I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value.
However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-every-commit test suite just exercises with a selection of values likely to expose obvious problems.
The fun part is people who criticize the generation of the integer triplets; yes, it's three nested loops and that's bad, but the total number of iterations will always be 16,777,216 no matter what algorithm you decide to use to generate them. So it uses nested loops since that's the most readable way to do it.
The point is being certain I haven't missed an edge case somewhere.
The hard part is the percentage rgb() values, of which there are technically an uncountably infinite number (since any real number in the range 0-100 is a legal percentage value). For those I generate all 16,777,216 integer values, and verify that converting to percentage and back yields the original value.
I don't see how that's a "tragedy", as in "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe".
You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even.
> Testing shows the presence, not the absence of bugs
If we're being pedantic, a tragedy is _a drama or literary work in which the main character is brought to ruin or suffers extreme sorrow, especially as a consequence of a tragic flaw, moral weakness, or inability to cope with unfavorable circumstances_.
If we're being pedantic, there's a second common definition of the word tragedy that does not refer to a drama or literary work, but "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe."
I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else with that time.
I am curious if there is any good way to measure that amount of waste. I agree that waste from extra and useless tests exists, but I am not convinced that waste is entirely bad.
It seems to me that waste is unavoidable with teams newer to automated testing and may simply be part of the cost of using automated tests. If that is the case then it seems better to compare the cost of bugs with no automated to cost of superfluous tests. In that comparison extra tests definitely seems like the lesser of two evils, even without hard numbers. I would prefer hard numbers because my intuition could be wrong.
I agree that test coverage does not ensure that the code is really tested. But not covered code is not tested code.
So i use coverage in this way, to spot not tested code.
You really shouldn't, or if you do it should be an extreme minority of tests and automated tests should do the heavy lifting.
Automated tests can do so much more than manual tests that shops still living by manual tests either have a damn good reason or are just as wrong as people who argued against Revision Control Systems (or people who argued for gotos instead of functions).
Automated tests will be executed identically each time, so no missing test cases because someone slacked off or made a typo. Automated test can serve as examples for how to use the code. Automated tests can aid in porting to new platforms, once it builds you can find all the bugs your care about swiftly. Automated test can be integrated with documentation, tools like doxyegn and mdbook make this easier.
Automated tests enable Continuous Integration. Are you familiar with Travis CI or Jenkins? If not, imagine a computer that a team commits their code to, instead of directly to the mainline Revision Control (Git master, svn head, etc...). That computer builds the software, runs all the tests, perhaps on every supported platform or in an environment very close to production, then only merges commits that appear to fully work. This doesn't completely eliminate bugs and broken builds, but the change is so large that teams without it are at a clear competitive disadvantage.
When integrated into process tests can be used to protect code from changes. If a test exercises an API and the team knows that is the purpose then when they change things in a way that break the test they shouldn't... This sounds vague or obvious, but consider this: At Facebook and Google they have a rule that if it is not tested new code doesn't have to care if it breaks. Both companies have team that make broad Sweeping changes. Facebook wrote a new std::string and Google use clangtools to make automated changes in thousands of places at once. Even if code breaks or APIs change as long as tests pass these people can be sure that they negatively impacted the product and are following their team's rules.
Automated Tests can... This list could go on for a very long time.
Maybe you're thinking about some specific applications/pieces of code? Probably not a video game?
I think it should be a mix. I disagree with the "extreme minority" portion for most projects. There are times where a manual test is the right answer and there's exploratory manual testing.
Obviously though running through a long sequence of testing manually for every code change is crazy. And some sort of CI setup like you describe is a must for every project this day and age.
Then there's also the question of unit tests vs. system/integration tests...
Why do you say most mutations will completely break an application? This is certainly not my experience.
Mutation testing systems normal use fairly stable operators (e.g changing a > to a >=). In most locations in the code changes such as these will have only a subtle effect.
If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes < or >= for example), and then runs your tests. If one test fails, your tests covered that statement.
It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim.
The other option is to accept that perfect coverage is not feasible, or possibly even a trap, and that you just need to cover the interesting bits.
I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code.
In the real world, you can't wait five minutes for the server to return a response. If the server returns the correct response after ten minutes, it's still wrong, unless the programmer has explicitly acknowledged that the procedure in question is a long-running procedure and has lengthened (but not eliminated) the time-out accordingly.
The halting problem is an irrelevant, Ivory-tower distraction in production code.
How would that be a false positive? If the substituted operator does not cause a test to fail, then your tests don't cover it. If the function of the program is not changed by changing the operator, then the operator does nothing and should be removed.
That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe that's not such a good idea after all. But that's a problem with all unit testing, and not just with mutation testing.
I was thinking that, for example, that a >= test is as valid as == in some cases. If == is actually used and is correct, substituting >= would not cause any valid test to fail. As I wrote this, however, I realized that if you substitute the inverse operator (!= for ==, < for >= etc.) and it is not caught, you can infer it is not covered. (edit: or that, in the context of the program as a whole, the result of the operation is irrelevant. I imagine this might legitimately happen, e.g. in the expansion of a macro or the instantiation of a template.)
Substituting == for >= should cause a test to fail. Either == is correct, or >=. They can't both be correct. Should the two things always be equal? Then they shouldn't be unequal. Is one allowed to be bigger than the other? Then that should be allowed, and not rejected.
If this change doesn't cause a test to fail, you're not testing edge cases for this comparison.
(Of course that's assuming you want perfect coverage.)
while ( j < JMAX) {
i = callback(j);
if( i >= FINAL) break;
...
If, in the specific circumstances of this use, callback(j) will always equal FINAL before it exceeds it, a test for equality here will not cause the program to behave differently.
The fallacy of your argument is that == is not an assertion that its arguments should always be equal; it is a test of whether they are at some specific point in the algorithm.
> even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases.
On the other hand, if you do cover all your inputs, you've covered all the cases regardless of what % code coverage or path coverage you have.
Has anyone seen a fuzzer that creates variants based on a test suite with 100% coverage? Hmm... the fuzzer still wouldn't necessarily know how to create the correct invariants. #lazyweb
The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definition of "compiled") language? Meh, you don't need to check every use of a variable at runtime to make sure the data types didn't go haywire.
The real Real Tragedy of 100% coverage is the number of shops who think they're done testing when they hit 100%. I've heard words to that effect out of the mouth of a test manager at Microsoft, as one example. No, code coverage is a metric, not the metric. Code coverage doesn't catch the bugs caused by the code you didn't write but should have, for example. Merely executing code is a simplistic test at best.
I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out.
Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them.
Anything past that is for companies with customers that need to maintain a certain level of quality and service. Worry about it when you get there.
Agreed. If you're a small start-up in a hurry and <4 devs, the best strategy would be to write top-down smoke tests on your API that will actually tell you when things break, even if not exactly where. Their actual derived coverage might be pretty high.
As long as that 10-20% is the more complex bits, and you are capable of picking out which bits are complex, it's IMO better to do that and spend the time saved on other shit that increases maintainability and reliability. Killing tech debt, killing your teammates, whatever.
This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.
Since what to test is a sliding scale, determining when and what to test is perhaps something that comes with exposure to good testers..
For me, a few references I use are "tests are a thinking help for specifying behavior", "tests hold behavior in place" and "test until you feel comfortable".
The first is a note that TDD thoughts are written by developers that write APIs, not by developers that write applications. TDD is a fantastic tool for an API designer, because they force you to think about the experience of using the API. So, whenever I design APIs, I like TDD. This is also a good argument for why you should minimize setup code that "goes behind the scenes" - if you're testing, say, a REST API, do as much of the setup and assertions as you can via the REST API as well!
The second helps me remember to think "a year from now, are all the behaviors this code needs to have obvious, or is someone likely to unintentionally break it?". I try to write tests that will flag if someone broke an important edge case - or the main use case! Tests can be used as the programmers equivalent of a carpenters' clamps, kind of.
The third is why I don't write as many tests anymore. I normally try to write one workflow-oriented feature test up front, like "When a user creates a new invoice, then that invoice should show up in the users list of invoices with the values they entered, plus x,y,z auto-generated parameters". As I implement the feature, if I come across a piece of logic that makes me feel uncomfortable - lots of branching, or code that's very important that it stays intact - I'll write a unit test or two to hold that in place; sometimes I don't write any unit tests, meaning I'd have written just one or two tests over the course of two or three days of implementation.
You should test the thing that makes you money first, and should delay testing supporting functionality. For instance, if I am writing a document converter, the thing that makes me money is the AST -> AST conversion. Testing that should come before testing parsing (bytes -> AST) and rendering (AST -> bytes).
The place where you make money is the place that will have the largest demand for new and changing functionality. And where things change the most is where you need tests to protect against regressions.
It varies wildly on the type of application you're building. I can only speak for front-end development of complex SPAs w/ React.
Generally, most architectures in this domain have a combination of UI components, a data store, a set of update logic for the data store, and a set of asynchronous controllers that respond to events, interact with APIs, and call the aforementioned update logic.
In React the UI components are declarative, they (generally) contain no logic or algorithms, just a mapping from state to DOM. I see basically zero value in testing these. Bugs are almost always of the 'forgot to actually implement' variety, or are related to the way the page is rendered in a particular browser, rather than the DOM output the components are responsible for.
The data store update logic is usually either simple setters/getters (which don't need testing) or complex data transformations (which do).
The controllers also come in simple and complex varieties. Simple ones (one API call, one data store update once it's resolved) don't need testing. Anything more complex than that probably does.
So those are the two main targets for testing in the apps I build. I generally don't bother with anything else.
There are exceptions though. For example, here's an accordion UI component I built which relies on an asynchronous manual DOM update after the React DOM update has finished resolving. This could almost definitely use tests, if only to help any maintenance developers understand what it's doing.
Basically, as long as you have some sort of sane architecture, there should only be a few potential targets for testing, and they should be easily identifiable.
Uncertainty. Look for the parts of the code where you feel you least sure you have the logic right.
A simple example of this is the bowling game kata: given the throws in a bowling game, calculate the (final) score. The 'hard' part is to keep strikes and spares in mind, including the bonus throws when a strike or spare is scored in the 10th frame.
If you were making an application that would help you keep track of your bowling score, that score calculator would have the highest ROI in terms of testing.
I've been working on a 1-man project with maybe 5% test coverage, just for some critical libraries that I ended up refactoring a few times. There's actually one little library that has 100% test coverage and super detailed error messages, because that's where many of the bugs seemed to happen.
I also have a simple integration test that just clicks through everything and makes sure nothing crashes.
Not having a lot of tests can be painful. Especially when you're learning new languages or frameworks, you almost always want to go back and rewrite some code (or just reorganize it into different files), and it's really nice to have tests when you do that. So sometimes that gives me the motivation to start writing up a bunch of tests, and then after that I dive in and refactor everything.
I mean I wouldn't settle for less than 200% test coverage on an automatic pilot for landing airplanes. If it is a one off script that happen to become a part of a temporary business process perhaps just a sample set of data, a desired output and small tool for comparing the results is enough.
I find that, as I'm building something from scratch, the vast majority of the errors I make are just things I didn't think of. Tests don't help there because I can't test on input that I don't even imagine happening. So I generally write few tests, because, to be honest, most code is trivial and algorithm-light. Sure, if I have to write a parser or something a bit more fiddly, I'll write a unit test to be sure that it's doing what I expect, but that tends to be the exception, not the rule. I do write my code with an eye toward later testability if it turns out to be necessary, but I find that to be fairly easy, and also a good measure of if I'm doing the write thing: most code that isn't testable is probably code that's difficult to read and maintain, anyway, so if I look at something and think "oof, how would I ever write a test for that?" I'll usually delete it and start over.
When I have something that should be working, I test it in a more functional/integrative manner, and move on.
Later, I'll write unit tests when I need to. If I want to refactor something, or drastically change the implementation of something, I'll write out some tests beforehand to be sure that the pre and post behaviors match.
I've always thought that TDD is just premature optimization. You're optimizing for the idea that you -- or someone -- will later need to make large enough changes to your code that you'd worry about breaking it. In my experience that's fairly rare, and you spend less time overall if you just write the tests as you need them, not up-front. Yes, writing a test when the code is fresh in your mind will be faster than writing it much later, but then you're writing a ton of test code that likely won't be necessary.
An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that they could write tests before changing it (if they deem it necessary). Anything else is just irresponsible.
(Note that I primarily work in strongly statically typed languages. If I were writing anything of complexity in ruby/python/JS/etc., I don't think I'd feel comfortable without testing a lot of things I'd consider trivial in other languages.)
(Also note that some things are just different: if you're writing a crypto library, then you absolutely need to write tests to verify behaviors, in part because you're building something that must conform to a formal spec, or else it's less than worthless.)
> An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that they could write tests before changing it (if they deem it necessary). Anything else is just irresponsible.
As someone who has had to fix plenty of legacy code, I have truly appreciated the people who have left me at least some working test suites to run - or just look at - and cursed many others. At the same time, I have generally been handed over code bases with tens or hundreds of thousands of lines, some of which had no useful tests.
If it is irresponsible to try to refactor or fix a codebase without first understanding all of it, it may be even more irresponsible to expect that those who follow in our footsteps will be able to do that, even if they are "gods of programming".
The reason why this is so was hammered home very strongly in Peter Naur's "Programming as Theory Building"[1].
Unless a code base is trivially small or simple, leaving it without meaningful tests instantly creates legacy code. I'll close with an excerpt from the back cover of "Working Effectively with Legacy Code[2].
> Is your code easy to change? Can you get nearly instantaneous feedback when you do change it? Do you understand it? If the answer to any of these questions is no, you have legacy code, and it is draining time and money away from your development efforts.
Yeah, I don't disagree with that, and one of the things I do before I hand off a code base to others is beef up test coverage somewhat, especially in places where the expected behavior of a bit of code might need to be codified to a certain extent.
Regarding your "legacy code" quotes: I don't write code that isn't easy to change, or easy to read. If I write code that's hard to read, I delete it and start again. If I absolutely cannot write something that's easy to read, I write tests around it and document the hell out of it (I'm iffy on comments and docs as well, because they _always_ end up out of date, and then are more of a hindrance than no comments at all).
I think I'm pretty good at what I do, but I wouldn't consider myself a "god of programming". Writing clear, concise code isn't hard. It really really really isn't. In my experience, the main blocker to that is ridiculous time lines and pressure to ship. I know it can be hard to push back against that pressure, but you owe it to yourself, future maintainers of your code, and the company you work for (even if the company doesn't realize or appreciate it at the time) to slow things down and do things the right way. A former colleague used to say, "It's not right because it works; it works because it's right". Just because a bit of code produces the output you want, it doesn't mean it's right. Write the right code -- readable, maintainable, verifiable, testable -- and you don't even need to worry about it working, because of course it will.
I lean on type systems heavily. If I were writing python or ruby, I'd have 10x as much test code as application code, because I just do not believe you can trust a dynamic/weakly-typed language without them. This is why I avoid such languages; I think any gains in rapid development that you get from such languages are quickly wiped out by the need to write extensive tests, or, lacking those, all the bugs that come up because you don't have them.
These days I write most things in scala, if I can. No, it's not a perfect language (honestly, I'd say a half to two thirds of it is crap, but the rest of it is amazing), but it has a strong type system that lets you lean on the compiler so much more than many other languages. Simply the fact that it compiles gives me much higher confidence than with most other languages.
My main issue with unit testing is what defines a unit?
Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression.
Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box.
That way tests actually assist with refactoring rather than be something that just exactly follows the code and breaks whenever a minor internal detail changes.
However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception.
I agree with you. That's called functional testing, and it is very useful, but it is not unit testing.
Unit testing: Test all methods and paths of a class, even private ones.
Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests.
ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.
- If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests.
- If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful.
- Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up.
I think any test strategy – unit, integration, e2e, acceptance, UI – pursued to exclusion is a bad idea. Different projects and different teams call for different balances between them.
> - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests.
I think we talk about the same thing, sometimes I will test something internal.
I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.
I'm kind of flexible on my terminology, perhaps more flexible than others, since my approach to testing has largely been self-taught from the experience of several shops.
To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue.
Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient.
I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.
"If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests."
This is actually where doctests work pretty well as they both document and test.
"If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful."
I don't find tracking down bugs with a repeatable test case to be much of a problem (events in live systems are a different story). It becomes even less of a problem if you sprinkle some assertions around the code that block off invalid code paths.
"Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up."
Building mocks in unit tests is usually even more cumbersome and tedious.
For overly large sets of real world data I've had some success taking live database snapshots and cutting out 95% of the data before using the cut down to size dump for testing.
I think that depends on the code base, I agree with your points for more procedural code bases, but for functional or OO code you should be able to test most of your logic without mocks or wiring too much up.
> Apprantly Kent beck meant unit to mean a single piece of behaviour
Which is just as ambiguous. Personally I stick with the single assert principal (which may be more than one literal assert) so that whatever I'm asserting is the behavior that this test is verifying.
You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members.
Now the use of the private member might lead to a different code path : it has to be tested.
Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked.
Functional Testing : test multiple unit at the same time to check the behavior of an entire functionality, that's why it is called functional testing. Stub the IO.
End-to-end testing: Test the entire system with the IO.
> it makes absolutely no sense to test private members
Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.
That's potentially one philosophy on it, sure. Yet that still doesn't preclude private methods from being an independently testable unit. Implementation details are the meat of the whole thing, and so they seem inherently testworthy. The glue code itself, less so.
Thanks for that thought though, definitely something I'll think on :)
You should be able to get the code coverage you need (even if 100%) by testing the public API of any class. If this isn't true, it means you have some private methods you can delete.
That's how the unit you're testing is used - through it's public API. The public API is the specification of how it works.
By only testing the public API, you allow yourself maximum ability to refactor in the future, while still maximizing code coverage. It means that simple refactoring (inlining methods, for example) won't break tests. More importantly, a failing test means something is wrong that is potentially relied upon elsewhere. If you test private methods, you will get test failures without the public API of the unit having changed at all.
Your point about "each function is a unit" is fine, you can justify testing private methods with that - but it's inefficient. If it's not necessary to be in the spec (public API), why have you made it so? You're over-complicating the design by locking yourself into implementation details in places where you don't need to.
In my experience I've found that testing private methods directly is a code smell. It shouldn't be necessary.
That's not potentially one philosophy, that's the definition of unit testing.
> Yet that still doesn't preclude private methods from being an independently testable unit
Then these methods ain't private as they break encapsulation. You can't have it both ways. Either a method is private or it isn't.
> Implementation details are the meat of the whole thing
But unit testing isn't about testing implementation details, it's about testing a specification that your API must respect, because that's the behavior collaborators consuming that API rely on. If your collaborators can call a private API then that API isn't private at first place.
I think the problem lies with the definition of "unit" which you quote above. As with designs stacking on top of one another at various levels, one level's functional test is often the next level's unit test. Your app's unit test might mock the very same things that are covered by the functional test for an underlying library. That library's unit tests might mock the very same things that are covered by the functional tests for one of its own dependencies, including the OS. And so on, even down to the level of functional units within a chip. (Chip verifiers have a lot to teach us software types about this kind of thing BTW, and I know because I've worked with a few.)
The distinction I always try to make is whether you're testing a contract or an implementation. If you're testing a contract, ROI is likely to remain high even as you move to finder granularities. If you're testing an implementation, so that the tests fail even when the implementation is 100% correct, then ROI falls off a cliff.
That's exactly what happens when the "unit" gets too small. If certain private methods can't be called a certain way because of constraints imposed by the class's public methods, then testing those calls is testing things that can't happen. That takes time away from testing (and implementing fixes for) things that can happen. If the class's contract changes so that those internal methods can be called in new ways, then yes, a 100% code-coverage unit test might catch the error. So, in all likelihood, would the new functional test accompanying the public-method change. The delta between the two, times the likelihood of such a scenario occurring in the first place, is too small to justify the cost of writing those tests and the likelihood that they'll generate false negatives.
>Unit testing: Test all methods and paths of a class, even private ones.
I disagree with this. Unit testing means testing the "unit" (i.e. the class/object) but still at a boundary level: the public API of that class. You should mock out any dependencies that object would have to ensure you are only testing that class but the class API has to be respected.
A private method is a hidden implementation detail, testing those would be overfitting your tests to the current implementation meaning if you change one character anywhere in your source you will almost certainly have to change one or more tests. Plus, if your tests are so tightly coupled to the implementation, it's likely to suffer from any bugs the implementation does causing them to be hidden (never forget that tests are also code and therefor have bugs at a similar rate to any other code).
Writing test code requires the same level (if not more) of engineering discipline that writing the code does.
>ROI of functional testing is high, as it is usually done with real data.
I also disagree with all of this. Functional testing is a kind of sanity check that the parts actually work together once assembled. If you have proper unit test coverage (and properly designed/engineered tests!) the functional testing is basically checking configuration. The problems with function testing are (1) testing is about checking code paths but which paths functional tests take can be hard to predict and differ between subsequent runs, making it hard to make any statement about what passing actually means. (2) is exactly what you mentioned as a positive: people tend to want to use what they call "real" data, i.e. data they have actually seen before. Which means it's probably only good for catching bugs they know about, not ones they've never seen before.
>In my opinion unit testing is a huge waste of time.
I would actually agree to this because I think answering the question of "how do we detect bad code" with "write more code" is problematic. I'd rather go the Haskell/Idris route and be able to prove that I have no bugs.
>Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.
This kind of response sounds like a self fulling prophecy. The exact point of mocking is to test code paths, i.e. if this dependency returns this result how will the class under inspection behave in response. Ideally you would use mocks to test everything that every dependency could respond. Unless the types involved have very few inhabitants, this isn't generally possible (even programatically) but the closer you get to this the more you can trust your tests.
> However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception.
An example of that exception: just yesterday I wrote tests for a serialization/deserialization utility, that translates between an object tree and a wire format used by our client. There was some tricky code around the "deserialize message into one of many possible objects" part, so I wrote a bunch of unit tests for the whole thing.
I find myself writing unit tests mostly for this kind of code - complex logic that transform data and/or execute "advanced" algorithms on it. I also write tests as close as possible to the boundaries of the complex logic - this way, when I'm sure the logic itself is sound, all integration bugs tend to be trivial to notice (it means someone fucked up the inputs or didn't handle outputs properly).
The more I think of it, the more I notice that I tend to structure my programs in a functional (as in functional programming) way - lots of "services" that take things as inputs and return them as outputs, without using any external state. So e.g. that serialization/deserialization service mentioned only takes a String as input, and returns a reference to a base-class object as an output (or the other way around for serialization). Making code conform to functional style makes it not only easier to test, it helps avoid some tests entirely.
> breaks whenever a minor internal detail changes.
This is usually where I draw the line for what to test. I don't care how a unit does what it does, I only care about what it does.
In my opinion a unit should have minimum side effects, none if possible, and the results should depend on the inputs. Write the test to the contract of what the caller should provide and caller will get back (and then any side that had to happening, like checking logs.)
If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written:
cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c &&
git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" &&
git push production master --force
When presented like this, relying on human intelligence and experience doesn't seem like such a bad thing after all.
Just so we're clear, my username was not inspired by this scheme.
Several other people are complaining about 100% coverage sometimes being misleading. One way to test your test is to randomly modify your code e.g. swapping a greater than for a less than. If your tests still pass, then they obviously missed this change in behaviour.
I've tried it a few times, I generally found it too slow to be an everyday part of my routine, but interesting tool to have in your belt for e.g. evaluating test in a codebase you didn't write yourself.
Have you ever tried it with one of the automated tools? I use Pitest extensively to mutate my Java code bases and found it is not too much slower then regular line and branch code coverage tools when you use a history file so only diffs are needed. The trade off of a few more seconds of build time is worth the benefit of the coverage report and being able to fail the build if coverage drops too low for me.
I feel like this high test coverage thing can only work if you have tight modules, tight interfaces, and you only bother testing at module boundaries. So the test cases almost function as a bit of executable API documentation - here's the method name, here's what it does, here's the contracts and/or static types, and.... given this input, you should get this output.
Do it for the high level bits you actually expose. If you're exposing everything, tests won't really save you - architecture and modularity are more fundamental and should be tackled first. If you're writing a big ball of mud, what benefit do you get testing a mudball?
One of the pressures for 100% coverage is working in a non-typesafe language. The gospel of coverage largely evolved in the Ruby community, where I often see test suites that look like a handrolled typechecker.
Unit tests are a poor substitute for correctness. Many unit tests does not a strong argument make.
Unit tests are typically inductive. Developer shows case A, B and C give the expected results for function f. God help us if our expectations are wrong. So, you're saying since A, B and C are correct therefore function f is correct. Well that may be, or maybe A, B and C are trivial cases, in other words, you've made a weak argument.
100% test coverage sounds like lazy management. Alas, the manager may have worked their way via social programming rather than computer programming. In such cases, better to say you have 110% test coverage.
This remembers me of recent projects where developers started to mock every piece of code.. The result was that all tests passed while the codebase exploded in real environments.
In my opinion the best advice is to force developers to use their brains. I know, there are a lot of sh*tty CTO/CEO/HoIT/SomeOther"Important"Position people out there seeing them as code monkeys and saying that developers are not paid to think but in that case the best thing developers could do is learn to say "NO"... My experience with that kind of people is that they need to learn the meaning of "NO" instead of wasting time and money in the end of the day.
I think unit testing makes sense when you have a function doing some math that can't be easily verified to be sensible by merely glancing at the code for two minutes.
I'm not sure there's much use for it in other scenarios.
There's something magical about being pretty sure things still work after any kind of refactor or new requirement.
I have been in situations like the author presents... decided not to test it and lo and behold it breaks production and causes $10,000 worth of revenue loss.
We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases
Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would.
I won't pretend that's a good decision for everyone. But I do think people take test-driven-development a little too religiously and often forget to ask themselves why they are writing a certain unit test.
I mean, before I was a manager I was a developer and I also went to a university where a professor once told me I had to unit test everything. But then, another professor told me to always use the singleton pattern. These days I view both statements as equally false.
I'm curious - how old are the systems you're working on?
In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression.
In other words: Even though it feels like it's slowing you down now, if you write tests while the functionality is still fresh in your mind, and you capture your assertions well, they'll pay off dividends in the future once you've forgotten what some block of code does.
I work in the public sector in Scandinavia and some of our oldest systems still in service run on an old tandem. So some of it is pretty old.
This gives us some unique abilities in terms of modeling our productivity of course, because we started measuring before anyone thought up unit testing.
Over the past 15 years, unit testing has failed to produce anything positive, and test driven development has been an absolute disaster.
That being said, this isn't something which will be universally true. A lot of the software we use isn't build by us, and I'm certainly that a lot of the suppliers on that software use unit tests quite extensively.
For the things we build ourselves, however, there has been almost no value in adopting modern test philosophies.
You say that systems have to be able to be worked on 5 years from now, but the truth is that most of our systems transport data and rarely live 5 years without getting rewritten to deliver better performance, higher levels of security or simply because the business has changed completely. A lot of it hold very few responsibilities as well, making it extremely obvious what to fix when a service breaks.
Don't get me wrong, we've seen problems we wouldn't have with 100% coverage. But that doesn't matter when spending resources fixing them is still a net positive on every account.
"This is based on the methodology that the more complex code is the more likely it is to have errors, hence the need for greater code coverage.
To better understand, let’s assume a scenario where you wanted to keep your change risk anti-patterns score below 30. To maintain this level with a code set that has a cyclomatic complexity of 10, you would need to achieve a code coverage (as defined by branch coverage) ratio of approximately 42%. If however, the complexity of your code was greater and you had a cyclomatic complexity of 20, you would need almost 72% code coverage if you wanted to maintain the same risk level.
Complexity increases risk. Testing decreases risk. The change risk anti-patterns score give you a metric to measure the correlation between the two."
345 comments
[ 3.3 ms ] story [ 105 ms ] threadBehaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected.
New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the standard "If your contribution doesn't have 100% coverage it won't be accepted". I don't think this is a bad default position.
So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually test any known edge cases.
Or one I've been personally hating lately, a nil pointer dereference in Golang from a well tested library that didn't do proper error propagation.
Yes. Code coverage should not be the primary measure of quality of your tests. That thinking leads to tests designed to cover lines, not use cases.
This is true.
But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)?
I think that a greater level of confidence in our code can be achieved by a combination of
- Judicious choices of unit and integration tests
- Static and dynamic analysis (if the language supports it), and
- Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases.
As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example.
If 100% coverage cannot be achieved, we need to ask ourselves:
- Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things?
- Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it?
- If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect.
- If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed.
References
[1]: https://en.m.wikipedia.org/wiki/QuickCheck
[2]: https://www.imperialviolet.org/2014/02/22/applebug.html
[3]: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...
More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x."
Test scenarios, not methods.
Let someone else look over your code, they might spot logic bugs that you oversee. If you write tests yourself, you will often oversee such logical bugs.
I have done (semi-)automatic reasoning and code verification with 100% code coverage of OS drivers. All the mathematical reasoning doesn't spot logical errors. You need more than one person to look over it.
I advocate spending time on identifying/inventing the correct abstractions over coverage.
Edit: while I'm a huge TDD advocate, I'm not a big advocate of measuring code coverage. That should only be necessary if you are trying to get a code base under coverage that wasn't TDD'd. Even then I'd rather add the coverage as I'm touching uncovered code. If it works and I'm not touching it, it doesn't need tests.
Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went untested. A single execution isn't even enough for me: I need a whole lot of testing on most of the code to be comfortable.
When I write Scala at work instead, I test algorithms, but a big percentage of my code is untested, and it all feels fine, because while not every piece of code that compiles works, the kind of bugs that I worry about are far smaller, especially if my code is type heavy, instead of building Map[String,Map[String,Int]] or anything like that. 100% code coverage in Scala rarely feels as valuable as in Ruby.
Also different styles make the value of having tests as a way to try to force good factoring changes by language and paradigm. Most functional Scala doesn't really need redesigning to make it easy to test: Functions without side effects are easy, and are easier to refactor. A deep Ruby inheritance tree with some unnecessary monkey patching just demands testing in comparison, and writing the tests themselves forces better design.
The author's code is Java, and there 95% of the reason for testing that isn't purely based on business requirements comes from runtime dependency injection systems that want you to put mutability everywhere. Those are reasons why 100% code coverage can still sell in a Java shop (I sure worked in some that used too many of the frameworks popular in the 00s), but in practice, there's many cases where the cost of the test is higher than the possible reward.
So if you ask me, whether 100% code coverage is a good idea or not depends a whole lot on your other tooling, and I think we should be moving towards situations where we want to write fewer tests.
I don't think I could write low-bug count code faster in statically typed scala than I could in unit tested ruby though. I mean I am well aware of what you're telling me, and it's obvious to me that having the compiler automatically check certain properties is a win. And yet, when push comes to shove, to get something done I'm more likely to reach for ruby.
It's something I've never come up with a good explanation for. Does static typing stunt prototyping and exploration? Do unit tests capture high level goals better?
I guess I don't agree with the assertion that ruby tests are useless. Because you test higher level things than you do with scala types.
Not in my experience. It is probably about what you personally find hard/uncomfortable/unfamiliar.
I've used static typing much more than dynamic. I'll admit confusion on things like ocaml polymorphic variants and haskell monads, nevertheless I wouldn't say I find static typing hard as a rule. But surely the point of static type checking is to constrict what you can do for safety and performance reasons. And surely the cost of that is you have less freedom - even when in the exploration phase
No. I didn't mean to imply or assert anything of the sort. Apologies if I inadvertently gave offense. You seem to be reading meanings into my reply that aren't there.
I don't know you from Adam. You asked a question in your post. I answered as best as I could.
That said, your latest statement
"I'll admit confusion on things like ocaml polymorphic variants and haskell monads,"
does seems to imply that you don't have much real world experience with static type systems in production (nothing wrong with that) since neither is an arcane concept or particularly difficult to understand.
If you haven't worked extensively with Haskell/Ocaml/SML etc, and you are extrapolating properties of 'static type systems' from those of Java or C++ then your idea of such type systems 'stunt prototyping and exploration' might make sense.
The rest of your comments are extrapolations from misunderstandings - not born of practical experience. My answer was based on (strictly) personal experience. Which is why I said "in my experience". I gladly concede that YMMV.
again, I was just answering your question in your original comment. I didn't mean to "imply" anything and used "probably" to mark my uncertainty about your real world experience with Ocaml/Haskell etc style static type systems.
I have extensive experience with both dynamically typed languages (mostly Python, Lua and Scheme) and statically typed languages (mostly Haskell and SML, besides Java). I answered out of my experience.
YMMV. And that is cool.
You're right, I haven't used ocaml or haskell in production. I did use F# though, which seems to be in the same ballpark as those languages in terms of having algebraic data types and inference and all the rest. I suppose the fact that I still don't really grok polymorphic variants after reading the real world ocaml a few times may say something about my ability, motivation, or at the very least how my brain is wired.
Fundamentally though, a type system is a restriction meant to help the programmer. This restriction must inevitably put you down a certain road when you explore stuff, right?
Initial "complete coverage" should probably start from mockups that test an entire API. The complete part should be that, in some way, the tests cover expected successes AND failures (successfully return failure) of every part of the API, but there's no need to test things individually if they've already been tested by other test cases.
Invariably reality will come up with more cases and someone will notice an area that wasn't quite fully tested. That's where a bug exists, but the golden test cases probably wouldn't have located it anyway. It'll take thousands or millions of users to hit that combination and notice it. Then you get to add another test case while you're fixing the problem.
But the developer response in a couple cases was to puff the code up with layers of fluff that just added levels of abstraction that just passed stuff down to the next layer, unchanged, with a bunch of parameter checking at each new level. This had the effect of adding a bunch of code with no chance of failure, artificially increasing the amount of code covered by the tests (which, by the way, were bullshit).
I got to rip all that junk out. It ran faster, was easier to understand and maintain, and I made sure I never, ever worked with the people who wrote that stuff.
https://news.ycombinator.com/item?id=14285926
To me, it's secondary and has tones of bike shedding in it. Writing tests is easy, mentally. Getting a good, simple, YAGNI/DRY architecture is more challenging and requires several iterations, something that is resisted if you have to rewrite all your unit tests. Let me put it this way, if you write an architecture that you later don't like, you would be more hesitant to scrap it and start over because all the additional work of the unit tests (especially if you're on a deadline). That's bad. To write a good architecture, a developer must be willing to realize he could have done it better and be willing to tear it down instead of build around the bad architecture. That's how you write 20 year code, which should be almost everyone's goal.
It is almost as if they see the success that we have had in dev/ops with the "cattle, not pets" philosophy, and want to apply that in their own field. Making the subject behave consistently and predictably, whether it is a machine or a human professional, would be a prerequisite for that.
Sure, the intention is good, but it promotes mindless repeating of patterns over thinking about what really helps.
Naming the test just "initialise" is not very useful as it doesn't assert what you expect the method under test to do. Given that the purpose of the initialise function is to populate a watchlists collection variable from the parameter, i'd name the test something like "initialise_daoRecordCountIs9_watchlistCountIs9". The pattern I generally use is <method_name>_<assertion_under_test>_<expected_result>.
Then, my test would be the following:
* Set up / mock the dao parameter to have 9 rows
* Create an instance of the class under test and push in the dao parameter
* Verify / Assert that the class under test now has 9 items in the watchlists variable - I'm assuming there is a public method to access that.
Just because you ran a function or ran a line doesn't mean it will work for the range of inputs you are allowing. If your function that you are running coverage on calls into the OS or a dependency, you also have to be ready for whatever that might return.
Therefore you can't tell if your code is right just by having run it. Worse, you might be lulled into a false sense of security by saying it works because that line is "covered by testing".
The real answer is to be smart, pick the right kind of testing at the right level to get the most bang for your buck. Unit test your complex logic. Stress test your locking, threading, perf, and io. Integration test your services.
When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself.
As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems
One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way.
Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.
I wonder what that would look like...
We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger tools should be able to hone in on areas where the result diverges from expectations.
And another example of something I'd want checked by the language is exception throwing / handling. It's another one of those places where code coverage won't help you unless you already know what you're looking for. Languages are getting better about it, but in general, handling errors is hard.
[0] https://developer.salesforce.com/docs/atlas.en-us.apexcode.m...
We can achieve this in dependently-typed languages like Idris. First we define a datatype 'Equal x y', which will represent the proposition that expression 'x' is equal to expression 'y':
There are two things to note:- There is only one way to construct a value of type `Equal x y`, which is to use the constructor we've called `Refl`.
- `Refl` only takes one argument, `x` (of arbitrary type `_`), and it only constructs values of type `Equal x x`. This is called "reflexivity", thus the name `Refl`.
Hence if we use the type `Expr x y` anywhere in our program, there is only one possible value we can provide that might typecheck (`Refl x`), and that will only typecheck if `Expr x x` unifies with `Expr x y`, which will only happen if `x` and `y` are the same thing; i.e. if they are equal. Thus the name `Equal`.
This `Equal` type comes in the standard library of all dependently typed languages, and is widely used. To use it for testing we just need to write a test, e.g. `myTest`, which returns some value indicating pass/fail, e.g. a `Boolean`. Then we can add the following to our program:
This will only type-check if `Equal myTest myTest` (the type of `Refl myTest`) unifies with `Equal myTest True`, which will only be the case if `myTest` evaluates to `True`.In haskell you embed domain specific languages when you require side effects:
This is like an interface specifying the basics of all stateful computations.You can use different implementations for production and testing without changing any code so mocking is built into everything.
http://dlang.org/spec/unittest.html
It's been very effective at improving the overall quality of the code. And because of CTFE (Compile Time Function Execution) and static assert, many code correctness tests can even be run at compile time.
On .NET it is a plain library, which requires the VS Ultimate editions to be useful and on Java it was mostly third party libraries.
C# design team is considering adding proper support as language feature, but it seems to be a very far away feature still.
C++20 might get contracts, but C++17 just got ratified, so who knows.
D, Ada 2012 and SPARK do already support contracts.
Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually:
1) fragile to implementation 2) opaque as to intent
Whereas input/output integration points are more reliable, transparent, and less fragile to implementation changes if the interface is maintained.
However, if you must do mock-level interaction testing, Spock has made it almost palatable in Javaland.
This is one area where functional fans get to make the imperative folks eat their lunch.
Usually because interfaces need to be clearly/well defined and overridden... In JS there are tools (rewire, proxyquire, etc) that can be used with tests in order to easily inject/replace dependencies without having to write code to support DI nearly as much. In fact, I'd say that there's usually no reason not to be very close to 100% coverage in JS projects.
It ties together related algorithms. You want "addDays(n)", "addHours(n)", "addMinutes(n)" and so on to be in the same class, even if they're one-liners.
I note that the examples you give have trivial test cases - presuming you don't care about overflow, and if you do, then those methods now have basis paths, whether explicit or implicit depends on the language.
"Pex uses a constraint solver to produce new test inputs which exercise different program behavior. "
So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/
[1]https://www.microsoft.com/en-us/research/publication/pex-whi...
[0] https://blogs.msdn.microsoft.com/visualstudio/2015/09/30/int...
To prove it works with all possible inputs, there are other tools at your disposal.
I'd like to hear about real world test scenarios where all possible inputs are tested.
The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches.
It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast).
See: https://github.com/dvyukov/go-fuzz#trophies and http://lcamtuf.coredump.cx/afl/
What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.
Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code.
Depending on your means and needs, you want to try both.
A sane type system would be a start for example. And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well.
TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs. But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.
[0] https://de.mathworks.com/products/polyspace.html
Brute force.
Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results.
Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field.
Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.
When you have one available, testing against the results against a reference implementation is a little more direct. Though, given how easy those property tests are, you might as well do both.
Some times I have, for example, matlab code that I 'know' is right and want to verify that my C/Python/Julia... code does the same thing. So then I just call the matlab code, call the new code and check if they're the same.
Or I have a slow exact method and want to see if my faster approximation always gives an answer 'close enough' to the correct answer.
This sounds pretty sketchy at first, but it turns out that almost all of the difficulty (and hence almost all of the bugs) are in getting the last few bits right, and the last few bits of a much-higher precision result don't matter when you're testing a lower-precision implementation, so combined with some basic sanity checks this is actually a very reasonable testing strategy (and widely used in industry).
However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-every-commit test suite just exercises with a selection of values likely to expose obvious problems.
https://github.com/ubernostrum/webcolors/blob/master/tests/f...
The fun part is people who criticize the generation of the integer triplets; yes, it's three nested loops and that's bad, but the total number of iterations will always be 16,777,216 no matter what algorithm you decide to use to generate them. So it uses nested loops since that's the most readable way to do it.
The hard part is the percentage rgb() values, of which there are technically an uncountably infinite number (since any real number in the range 0-100 is a legal percentage value). For those I generate all 16,777,216 integer values, and verify that converting to percentage and back yields the original value.
You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even.
> Testing shows the presence, not the absence of bugs
Dijkstra (1969)
Definitely not a tragedy.
I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else with that time.
It seems to me that waste is unavoidable with teams newer to automated testing and may simply be part of the cost of using automated tests. If that is the case then it seems better to compare the cost of bugs with no automated to cost of superfluous tests. In that comparison extra tests definitely seems like the lesser of two evils, even without hard numbers. I would prefer hard numbers because my intuition could be wrong.
1. Take the number of hours spent writing tests.
2. Multiply by whatever percentage of the tests are unnecessary.
3. Multiply by the labor cost per hour. Or revenue that was not made (e.g. if you could have billed those hours to a customer, but didn't).
The resulting number could either be a big deal or not, depending on how big your organization is and how much time you spent on superfluous tests.
Automated tests can do so much more than manual tests that shops still living by manual tests either have a damn good reason or are just as wrong as people who argued against Revision Control Systems (or people who argued for gotos instead of functions).
Automated tests will be executed identically each time, so no missing test cases because someone slacked off or made a typo. Automated test can serve as examples for how to use the code. Automated tests can aid in porting to new platforms, once it builds you can find all the bugs your care about swiftly. Automated test can be integrated with documentation, tools like doxyegn and mdbook make this easier.
Automated tests enable Continuous Integration. Are you familiar with Travis CI or Jenkins? If not, imagine a computer that a team commits their code to, instead of directly to the mainline Revision Control (Git master, svn head, etc...). That computer builds the software, runs all the tests, perhaps on every supported platform or in an environment very close to production, then only merges commits that appear to fully work. This doesn't completely eliminate bugs and broken builds, but the change is so large that teams without it are at a clear competitive disadvantage.
When integrated into process tests can be used to protect code from changes. If a test exercises an API and the team knows that is the purpose then when they change things in a way that break the test they shouldn't... This sounds vague or obvious, but consider this: At Facebook and Google they have a rule that if it is not tested new code doesn't have to care if it breaks. Both companies have team that make broad Sweeping changes. Facebook wrote a new std::string and Google use clangtools to make automated changes in thousands of places at once. Even if code breaks or APIs change as long as tests pass these people can be sure that they negatively impacted the product and are following their team's rules.
Automated Tests can... This list could go on for a very long time.
I think it should be a mix. I disagree with the "extreme minority" portion for most projects. There are times where a manual test is the right answer and there's exploratory manual testing.
Obviously though running through a long sequence of testing manually for every code change is crazy. And some sort of CI setup like you describe is a must for every project this day and age.
Then there's also the question of unit tests vs. system/integration tests...
Why not prove the code is correct instead? Should be much cheaper than 100% coverage and more certain.
Mutation testing systems normal use fairly stable operators (e.g changing a > to a >=). In most locations in the code changes such as these will have only a subtle effect.
It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim.
The other option is to accept that perfect coverage is not feasible, or possibly even a trap, and that you just need to cover the interesting bits.
http://pitest.org/faq/
The halting problem is an irrelevant, Ivory-tower distraction in production code.
That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe that's not such a good idea after all. But that's a problem with all unit testing, and not just with mutation testing.
If this change doesn't cause a test to fail, you're not testing edge cases for this comparison.
(Of course that's assuming you want perfect coverage.)
The fallacy of your argument is that == is not an assertion that its arguments should always be equal; it is a test of whether they are at some specific point in the algorithm.
On the other hand, if you do cover all your inputs, you've covered all the cases regardless of what % code coverage or path coverage you have.
The real Real Tragedy of 100% coverage is the number of shops who think they're done testing when they hit 100%. I've heard words to that effect out of the mouth of a test manager at Microsoft, as one example. No, code coverage is a metric, not the metric. Code coverage doesn't catch the bugs caused by the code you didn't write but should have, for example. Merely executing code is a simplistic test at best.
Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them.
Anything past that is for companies with customers that need to maintain a certain level of quality and service. Worry about it when you get there.
This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.
For me, a few references I use are "tests are a thinking help for specifying behavior", "tests hold behavior in place" and "test until you feel comfortable".
The first is a note that TDD thoughts are written by developers that write APIs, not by developers that write applications. TDD is a fantastic tool for an API designer, because they force you to think about the experience of using the API. So, whenever I design APIs, I like TDD. This is also a good argument for why you should minimize setup code that "goes behind the scenes" - if you're testing, say, a REST API, do as much of the setup and assertions as you can via the REST API as well!
The second helps me remember to think "a year from now, are all the behaviors this code needs to have obvious, or is someone likely to unintentionally break it?". I try to write tests that will flag if someone broke an important edge case - or the main use case! Tests can be used as the programmers equivalent of a carpenters' clamps, kind of.
The third is why I don't write as many tests anymore. I normally try to write one workflow-oriented feature test up front, like "When a user creates a new invoice, then that invoice should show up in the users list of invoices with the values they entered, plus x,y,z auto-generated parameters". As I implement the feature, if I come across a piece of logic that makes me feel uncomfortable - lots of branching, or code that's very important that it stays intact - I'll write a unit test or two to hold that in place; sometimes I don't write any unit tests, meaning I'd have written just one or two tests over the course of two or three days of implementation.
The place where you make money is the place that will have the largest demand for new and changing functionality. And where things change the most is where you need tests to protect against regressions.
Generally, most architectures in this domain have a combination of UI components, a data store, a set of update logic for the data store, and a set of asynchronous controllers that respond to events, interact with APIs, and call the aforementioned update logic.
In React the UI components are declarative, they (generally) contain no logic or algorithms, just a mapping from state to DOM. I see basically zero value in testing these. Bugs are almost always of the 'forgot to actually implement' variety, or are related to the way the page is rendered in a particular browser, rather than the DOM output the components are responsible for.
The data store update logic is usually either simple setters/getters (which don't need testing) or complex data transformations (which do).
The controllers also come in simple and complex varieties. Simple ones (one API call, one data store update once it's resolved) don't need testing. Anything more complex than that probably does.
So those are the two main targets for testing in the apps I build. I generally don't bother with anything else.
There are exceptions though. For example, here's an accordion UI component I built which relies on an asynchronous manual DOM update after the React DOM update has finished resolving. This could almost definitely use tests, if only to help any maintenance developers understand what it's doing.
https://gist.github.com/JonathonAshworth/b401810b965149348d0...
Basically, as long as you have some sort of sane architecture, there should only be a few potential targets for testing, and they should be easily identifiable.
A simple example of this is the bowling game kata: given the throws in a bowling game, calculate the (final) score. The 'hard' part is to keep strikes and spares in mind, including the bonus throws when a strike or spare is scored in the 10th frame.
If you were making an application that would help you keep track of your bowling score, that score calculator would have the highest ROI in terms of testing.
I also have a simple integration test that just clicks through everything and makes sure nothing crashes.
Not having a lot of tests can be painful. Especially when you're learning new languages or frameworks, you almost always want to go back and rewrite some code (or just reorganize it into different files), and it's really nice to have tests when you do that. So sometimes that gives me the motivation to start writing up a bunch of tests, and then after that I dive in and refactor everything.
I mean I wouldn't settle for less than 200% test coverage on an automatic pilot for landing airplanes. If it is a one off script that happen to become a part of a temporary business process perhaps just a sample set of data, a desired output and small tool for comparing the results is enough.
When I have something that should be working, I test it in a more functional/integrative manner, and move on.
Later, I'll write unit tests when I need to. If I want to refactor something, or drastically change the implementation of something, I'll write out some tests beforehand to be sure that the pre and post behaviors match.
I've always thought that TDD is just premature optimization. You're optimizing for the idea that you -- or someone -- will later need to make large enough changes to your code that you'd worry about breaking it. In my experience that's fairly rare, and you spend less time overall if you just write the tests as you need them, not up-front. Yes, writing a test when the code is fresh in your mind will be faster than writing it much later, but then you're writing a ton of test code that likely won't be necessary.
An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that they could write tests before changing it (if they deem it necessary). Anything else is just irresponsible.
(Note that I primarily work in strongly statically typed languages. If I were writing anything of complexity in ruby/python/JS/etc., I don't think I'd feel comfortable without testing a lot of things I'd consider trivial in other languages.)
(Also note that some things are just different: if you're writing a crypto library, then you absolutely need to write tests to verify behaviors, in part because you're building something that must conform to a formal spec, or else it's less than worthless.)
As someone who has had to fix plenty of legacy code, I have truly appreciated the people who have left me at least some working test suites to run - or just look at - and cursed many others. At the same time, I have generally been handed over code bases with tens or hundreds of thousands of lines, some of which had no useful tests.
If it is irresponsible to try to refactor or fix a codebase without first understanding all of it, it may be even more irresponsible to expect that those who follow in our footsteps will be able to do that, even if they are "gods of programming".
The reason why this is so was hammered home very strongly in Peter Naur's "Programming as Theory Building"[1].
Unless a code base is trivially small or simple, leaving it without meaningful tests instantly creates legacy code. I'll close with an excerpt from the back cover of "Working Effectively with Legacy Code[2].
> Is your code easy to change? Can you get nearly instantaneous feedback when you do change it? Do you understand it? If the answer to any of these questions is no, you have legacy code, and it is draining time and money away from your development efforts.
[1]: https://news.ycombinator.com/item?id=10833278
[2]: https://www.amazon.com/Working-Effectively-Legacy-Robert-Mar...
Regarding your "legacy code" quotes: I don't write code that isn't easy to change, or easy to read. If I write code that's hard to read, I delete it and start again. If I absolutely cannot write something that's easy to read, I write tests around it and document the hell out of it (I'm iffy on comments and docs as well, because they _always_ end up out of date, and then are more of a hindrance than no comments at all).
I think I'm pretty good at what I do, but I wouldn't consider myself a "god of programming". Writing clear, concise code isn't hard. It really really really isn't. In my experience, the main blocker to that is ridiculous time lines and pressure to ship. I know it can be hard to push back against that pressure, but you owe it to yourself, future maintainers of your code, and the company you work for (even if the company doesn't realize or appreciate it at the time) to slow things down and do things the right way. A former colleague used to say, "It's not right because it works; it works because it's right". Just because a bit of code produces the output you want, it doesn't mean it's right. Write the right code -- readable, maintainable, verifiable, testable -- and you don't even need to worry about it working, because of course it will.
I lean on type systems heavily. If I were writing python or ruby, I'd have 10x as much test code as application code, because I just do not believe you can trust a dynamic/weakly-typed language without them. This is why I avoid such languages; I think any gains in rapid development that you get from such languages are quickly wiped out by the need to write extensive tests, or, lacking those, all the bugs that come up because you don't have them.
These days I write most things in scala, if I can. No, it's not a perfect language (honestly, I'd say a half to two thirds of it is crap, but the rest of it is amazing), but it has a strong type system that lets you lean on the compiler so much more than many other languages. Simply the fact that it compiles gives me much higher confidence than with most other languages.
Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression.
Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box.
That way tests actually assist with refactoring rather than be something that just exactly follows the code and breaks whenever a minor internal detail changes.
However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception.
Unit testing: Test all methods and paths of a class, even private ones.
Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests.
ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.
- If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests.
- If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful.
- Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up.
I think any test strategy – unit, integration, e2e, acceptance, UI – pursued to exclusion is a bad idea. Different projects and different teams call for different balances between them.
I think we talk about the same thing, sometimes I will test something internal.
I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.
To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue.
Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient.
I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.
This is actually where doctests work pretty well as they both document and test.
"If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful."
I don't find tracking down bugs with a repeatable test case to be much of a problem (events in live systems are a different story). It becomes even less of a problem if you sprinkle some assertions around the code that block off invalid code paths.
"Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up."
Building mocks in unit tests is usually even more cumbersome and tedious.
For overly large sets of real world data I've had some success taking live database snapshots and cutting out 95% of the data before using the cut down to size dump for testing.
Which is just as ambiguous. Personally I stick with the single assert principal (which may be more than one literal assert) so that whatever I'm asserting is the behavior that this test is verifying.
You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members.
Now the use of the private member might lead to a different code path : it has to be tested.
Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked.
Functional Testing : test multiple unit at the same time to check the behavior of an entire functionality, that's why it is called functional testing. Stub the IO.
End-to-end testing: Test the entire system with the IO.
Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.
Here's the reason:
We're not writing tests; we're writing SPECIFICATIONS.
Private methods are implementation details; they're not part of the specification.
Thanks for that thought though, definitely something I'll think on :)
That's how the unit you're testing is used - through it's public API. The public API is the specification of how it works.
By only testing the public API, you allow yourself maximum ability to refactor in the future, while still maximizing code coverage. It means that simple refactoring (inlining methods, for example) won't break tests. More importantly, a failing test means something is wrong that is potentially relied upon elsewhere. If you test private methods, you will get test failures without the public API of the unit having changed at all.
Your point about "each function is a unit" is fine, you can justify testing private methods with that - but it's inefficient. If it's not necessary to be in the spec (public API), why have you made it so? You're over-complicating the design by locking yourself into implementation details in places where you don't need to.
In my experience I've found that testing private methods directly is a code smell. It shouldn't be necessary.
That's not potentially one philosophy, that's the definition of unit testing.
> Yet that still doesn't preclude private methods from being an independently testable unit
Then these methods ain't private as they break encapsulation. You can't have it both ways. Either a method is private or it isn't.
> Implementation details are the meat of the whole thing
But unit testing isn't about testing implementation details, it's about testing a specification that your API must respect, because that's the behavior collaborators consuming that API rely on. If your collaborators can call a private API then that API isn't private at first place.
The distinction I always try to make is whether you're testing a contract or an implementation. If you're testing a contract, ROI is likely to remain high even as you move to finder granularities. If you're testing an implementation, so that the tests fail even when the implementation is 100% correct, then ROI falls off a cliff.
That's exactly what happens when the "unit" gets too small. If certain private methods can't be called a certain way because of constraints imposed by the class's public methods, then testing those calls is testing things that can't happen. That takes time away from testing (and implementing fixes for) things that can happen. If the class's contract changes so that those internal methods can be called in new ways, then yes, a 100% code-coverage unit test might catch the error. So, in all likelihood, would the new functional test accompanying the public-method change. The delta between the two, times the likelihood of such a scenario occurring in the first place, is too small to justify the cost of writing those tests and the likelihood that they'll generate false negatives.
I disagree with this. Unit testing means testing the "unit" (i.e. the class/object) but still at a boundary level: the public API of that class. You should mock out any dependencies that object would have to ensure you are only testing that class but the class API has to be respected.
A private method is a hidden implementation detail, testing those would be overfitting your tests to the current implementation meaning if you change one character anywhere in your source you will almost certainly have to change one or more tests. Plus, if your tests are so tightly coupled to the implementation, it's likely to suffer from any bugs the implementation does causing them to be hidden (never forget that tests are also code and therefor have bugs at a similar rate to any other code).
Writing test code requires the same level (if not more) of engineering discipline that writing the code does.
>ROI of functional testing is high, as it is usually done with real data.
I also disagree with all of this. Functional testing is a kind of sanity check that the parts actually work together once assembled. If you have proper unit test coverage (and properly designed/engineered tests!) the functional testing is basically checking configuration. The problems with function testing are (1) testing is about checking code paths but which paths functional tests take can be hard to predict and differ between subsequent runs, making it hard to make any statement about what passing actually means. (2) is exactly what you mentioned as a positive: people tend to want to use what they call "real" data, i.e. data they have actually seen before. Which means it's probably only good for catching bugs they know about, not ones they've never seen before.
>In my opinion unit testing is a huge waste of time.
I would actually agree to this because I think answering the question of "how do we detect bad code" with "write more code" is problematic. I'd rather go the Haskell/Idris route and be able to prove that I have no bugs.
>Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.
This kind of response sounds like a self fulling prophecy. The exact point of mocking is to test code paths, i.e. if this dependency returns this result how will the class under inspection behave in response. Ideally you would use mocks to test everything that every dependency could respond. Unless the types involved have very few inhabitants, this isn't generally possible (even programatically) but the closer you get to this the more you can trust your tests.
An example of that exception: just yesterday I wrote tests for a serialization/deserialization utility, that translates between an object tree and a wire format used by our client. There was some tricky code around the "deserialize message into one of many possible objects" part, so I wrote a bunch of unit tests for the whole thing.
I find myself writing unit tests mostly for this kind of code - complex logic that transform data and/or execute "advanced" algorithms on it. I also write tests as close as possible to the boundaries of the complex logic - this way, when I'm sure the logic itself is sound, all integration bugs tend to be trivial to notice (it means someone fucked up the inputs or didn't handle outputs properly).
The more I think of it, the more I notice that I tend to structure my programs in a functional (as in functional programming) way - lots of "services" that take things as inputs and return them as outputs, without using any external state. So e.g. that serialization/deserialization service mentioned only takes a String as input, and returns a reference to a base-class object as an output (or the other way around for serialization). Making code conform to functional style makes it not only easier to test, it helps avoid some tests entirely.
This is usually where I draw the line for what to test. I don't care how a unit does what it does, I only care about what it does.
In my opinion a unit should have minimum side effects, none if possible, and the results should depend on the inputs. Write the test to the contract of what the caller should provide and caller will get back (and then any side that had to happening, like checking logs.)
Just so we're clear, my username was not inspired by this scheme.
This is known as mutation testing.
https://en.wikipedia.org/wiki/Mutation_testing
I've tried it a few times, I generally found it too slow to be an everyday part of my routine, but interesting tool to have in your belt for e.g. evaluating test in a codebase you didn't write yourself.
http://pitest.org/quickstart/maven/
Do it for the high level bits you actually expose. If you're exposing everything, tests won't really save you - architecture and modularity are more fundamental and should be tackled first. If you're writing a big ball of mud, what benefit do you get testing a mudball?
Unit tests are typically inductive. Developer shows case A, B and C give the expected results for function f. God help us if our expectations are wrong. So, you're saying since A, B and C are correct therefore function f is correct. Well that may be, or maybe A, B and C are trivial cases, in other words, you've made a weak argument.
100% test coverage sounds like lazy management. Alas, the manager may have worked their way via social programming rather than computer programming. In such cases, better to say you have 110% test coverage.
In my opinion the best advice is to force developers to use their brains. I know, there are a lot of sh*tty CTO/CEO/HoIT/SomeOther"Important"Position people out there seeing them as code monkeys and saying that developers are not paid to think but in that case the best thing developers could do is learn to say "NO"... My experience with that kind of people is that they need to learn the meaning of "NO" instead of wasting time and money in the end of the day.
I'm not sure there's much use for it in other scenarios.
There's something magical about being pretty sure things still work after any kind of refactor or new requirement.
I have been in situations like the author presents... decided not to test it and lo and behold it breaks production and causes $10,000 worth of revenue loss.
Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would.
I won't pretend that's a good decision for everyone. But I do think people take test-driven-development a little too religiously and often forget to ask themselves why they are writing a certain unit test.
I mean, before I was a manager I was a developer and I also went to a university where a professor once told me I had to unit test everything. But then, another professor told me to always use the singleton pattern. These days I view both statements as equally false.
In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression.
In other words: Even though it feels like it's slowing you down now, if you write tests while the functionality is still fresh in your mind, and you capture your assertions well, they'll pay off dividends in the future once you've forgotten what some block of code does.
This gives us some unique abilities in terms of modeling our productivity of course, because we started measuring before anyone thought up unit testing.
Over the past 15 years, unit testing has failed to produce anything positive, and test driven development has been an absolute disaster.
That being said, this isn't something which will be universally true. A lot of the software we use isn't build by us, and I'm certainly that a lot of the suppliers on that software use unit tests quite extensively.
For the things we build ourselves, however, there has been almost no value in adopting modern test philosophies.
You say that systems have to be able to be worked on 5 years from now, but the truth is that most of our systems transport data and rarely live 5 years without getting rewritten to deliver better performance, higher levels of security or simply because the business has changed completely. A lot of it hold very few responsibilities as well, making it extremely obvious what to fix when a service breaks.
Don't get me wrong, we've seen problems we wouldn't have with 100% coverage. But that doesn't matter when spending resources fixing them is still a net positive on every account.
IntelliJ has a plugin
http://www.ncover.com/blog/change-risk-anti-patterns-code-co...
"This is based on the methodology that the more complex code is the more likely it is to have errors, hence the need for greater code coverage.
To better understand, let’s assume a scenario where you wanted to keep your change risk anti-patterns score below 30. To maintain this level with a code set that has a cyclomatic complexity of 10, you would need to achieve a code coverage (as defined by branch coverage) ratio of approximately 42%. If however, the complexity of your code was greater and you had a cyclomatic complexity of 20, you would need almost 72% code coverage if you wanted to maintain the same risk level.
Complexity increases risk. Testing decreases risk. The change risk anti-patterns score give you a metric to measure the correlation between the two."