I guess I don't have any need to listen to that Viktor guy, then. If Viktor runs his tests, gets 50%, and wants to try and convince me it doesn't mean anything, he's going to find it falling on deaf ears. I don't care how "meaningful" ones tests are, if they only cover half the code someone is queueing for a bad day somewhere down the line.
OTOH, I have heard with my own ear holes a test manager say that they had hit 100% coverage and therefore testing is complete (paraphrasing). A test manager at Microsoft said this in a meeting with other people present. So I can see where Viktor might be coming from, but he has still thrown out the baby with the bathwater.
I don't know everyone else's experience, but I found unit testing encouraged me to write better quality code. Testable code is better code. Methods are smaller and have more focused purpose, leading to more reusability. Writing unit tests with >98% coverage forces me to think more deeply about and have a greater understanding of the code I'm writing and how it all interacts. I am more likely to consider edge cases where things might go awry and deal with them. So even if someone believes that tests don't buy them anything directly, I can only speak for myself but the second and third order effects have been great for my career.
I found that unit testing with small methods meant I was writing my tests to those methods.
But the methods were specific to one implementation approach.
If I wanted to rewrite the algorithm, eg, for clarity or performance, I ended up replacing many of the tests, which felt like test churn.
I prefer trying to figure out where the semi-permanent/API-like interface are in the code, and test through that. The result is closer to integration testing than unit testing, although I use a unit test testing framework for it.
I had the opposite experience. One of the many good practices unit testing encourages is encapsulation. With encapsulation and mocking during testing, there is very little chance of a cascade affect when changing a method or doing some refactoring.
If you only change the internals of a method, then it's clear that the method call is the API.
But the specific method calls are rarely the deliverable (outside of library packages). Those are usually much higher, eg, web API or command-line options. Instead, the methods can be rather fluid.
My experience is that the "unit testing with small methods" approach usually emphasizes testing each method.
The analogy I use is that of building a bridge, where you don't care what sort of bridge is made, only that it can handle a specific amount of traffic.
So I build a truss bridge, with tests that each of the struts is correct.
But this is programming, and I decide it's better to replace the truss with a suspension bridge, so I do that.
Which means I have to throw away all the strut tests, since I don't have them.
While if I had only used load tests from the start - integration tests for what the customer actually wants - then I could do wholesale rewrites without changing much of the tests.
I want to be clear I'm not disagreeing with you here. But I would like to point out if you were building a real bridge every component would be inspected individually. They wouldn't just let you build a whole bridge and do a final load test. Whether that level of robustness/ confidence is important will be highly contextual.
Yes, and part of the mythos of unit testing - especially around small methods - is based on that real-world analogy, when "this is programming", and many of the rules are quite different.
As another analogy, if you breadboard a device you can test each individual component directly. Once it's on a chip, most of those individual tests can no longer be used. That's not to say they can't be tested, only that the testing must be done on a higher interface level.
So, do you use those higher-level tests, which are more like intergration tests, or do you write the low-level unit tests and expect to throw the away at any sort of non-trivial code changes?
> "Viktor Fracic: “Code coverage means nothing." I guess I don't have any need to listen to that Viktor guy, then.
The full quote is:
"Viktor Fracic: “Code coverage means nothing. I don’t understand why people are still obsessed with code coverage. It’s not really about how many tests you have, but the quality of those tests. I’ve seen companies that have close to 100% code coverage but have meaningless tests that don’t prove anything."
This article goes on about "why traditional testing is not sufficient". I think their view of 'traditional testing' is a strawman.
Traditional testing never said that 100% code testing was sufficient.
Even 100% branch testing isn't sufficient. SQLite gets bug reports (eg, through fuzzing) even though they have some of the highest covered code ever.
Viktor Fracic's comment “Code coverage means nothing" is the traditional view on coverage. Code coverage only means something when you are testing so that code coverage can help. Otherwise, it's easy to get high coverage with meaningness tests.
And that's been the view of code coverage for a very long time.
Ditto for "High quality testing takes time and doesn’t guarantee success." Has anyone ever said that high-quality testing guarantees success? Perhaps a few odd-balls, but it's never been the majority view.
"I’d rather see people move fast and be comfortable with failing" depends very much on your market. (Boeing avionics, ahem.)
In other words, testing has never been sufficient. Only necessary.
Seems like they are essentially arguing for the ol' "move fast and break things" - though I guess in a slightly more responsible manner: You can't catch all problems with thests, so it's important to be able to react quickly to problems - and in fact, being able to react quickly is already important from a business perspective anyway.
Sounds reasonable, I just wonder what kind of practical differences this advice does make.
I do hope this isn't a lot of talk to go back to the Facebook style of "moving fast and breaking things" in the end. The "let your users be the QA" part somewhat sounds like that...
13 comments
[ 2.8 ms ] story [ 38.3 ms ] threadI guess I don't have any need to listen to that Viktor guy, then. If Viktor runs his tests, gets 50%, and wants to try and convince me it doesn't mean anything, he's going to find it falling on deaf ears. I don't care how "meaningful" ones tests are, if they only cover half the code someone is queueing for a bad day somewhere down the line.
OTOH, I have heard with my own ear holes a test manager say that they had hit 100% coverage and therefore testing is complete (paraphrasing). A test manager at Microsoft said this in a meeting with other people present. So I can see where Viktor might be coming from, but he has still thrown out the baby with the bathwater.
But the methods were specific to one implementation approach.
If I wanted to rewrite the algorithm, eg, for clarity or performance, I ended up replacing many of the tests, which felt like test churn.
I prefer trying to figure out where the semi-permanent/API-like interface are in the code, and test through that. The result is closer to integration testing than unit testing, although I use a unit test testing framework for it.
If you only change the internals of a method, then it's clear that the method call is the API.
But the specific method calls are rarely the deliverable (outside of library packages). Those are usually much higher, eg, web API or command-line options. Instead, the methods can be rather fluid.
My experience is that the "unit testing with small methods" approach usually emphasizes testing each method.
The analogy I use is that of building a bridge, where you don't care what sort of bridge is made, only that it can handle a specific amount of traffic.
So I build a truss bridge, with tests that each of the struts is correct.
But this is programming, and I decide it's better to replace the truss with a suspension bridge, so I do that.
Which means I have to throw away all the strut tests, since I don't have them.
While if I had only used load tests from the start - integration tests for what the customer actually wants - then I could do wholesale rewrites without changing much of the tests.
As another analogy, if you breadboard a device you can test each individual component directly. Once it's on a chip, most of those individual tests can no longer be used. That's not to say they can't be tested, only that the testing must be done on a higher interface level.
So, do you use those higher-level tests, which are more like intergration tests, or do you write the low-level unit tests and expect to throw the away at any sort of non-trivial code changes?
The full quote is:
"Viktor Fracic: “Code coverage means nothing. I don’t understand why people are still obsessed with code coverage. It’s not really about how many tests you have, but the quality of those tests. I’ve seen companies that have close to 100% code coverage but have meaningless tests that don’t prove anything."
Traditional testing never said that 100% code testing was sufficient.
Even 100% branch testing isn't sufficient. SQLite gets bug reports (eg, through fuzzing) even though they have some of the highest covered code ever.
Viktor Fracic's comment “Code coverage means nothing" is the traditional view on coverage. Code coverage only means something when you are testing so that code coverage can help. Otherwise, it's easy to get high coverage with meaningness tests.
And that's been the view of code coverage for a very long time.
Ditto for "High quality testing takes time and doesn’t guarantee success." Has anyone ever said that high-quality testing guarantees success? Perhaps a few odd-balls, but it's never been the majority view.
"I’d rather see people move fast and be comfortable with failing" depends very much on your market. (Boeing avionics, ahem.)
In other words, testing has never been sufficient. Only necessary.
that coverage as a metric is insufficient to quantify the quality of testing is not a surprise to anyone.
Sounds reasonable, I just wonder what kind of practical differences this advice does make.
I do hope this isn't a lot of talk to go back to the Facebook style of "moving fast and breaking things" in the end. The "let your users be the QA" part somewhat sounds like that...