I was wondering if people are still using test coverage reports, and if so, which tools and whether the reports are aggregated somewhere (on premise or in the cloud).
Absolutely not. It is hard enough to get developers writing meaningful tests that exercise edge cases to make code hard to break. Far too often they do it because "you should" and "we have a policy". Coverage reports too easily get people focusing on numbers and policy over understanding how to identify and strengthen key functionality so it's hard to break as things evolve.
What you end up with is:
1. Tests that at least invoke code that may be overlooked, but due to mocks, invoke it in a way thats meaningless.
2. Twice as much code to update when doing any feature work, refactoring, ... or even bug fixing!
3. A false sense of security because "tests passed!"
4. A lot of wasted time writing tests for code you end up either throwing out or significantly refactoring before it ever becomes a remotely active part of the project.
I'm jaded.
I LOVE good tests, but far, far too often they get in the way of real progress that actually matters to an end user. Tests are supposed to be worth their weight in gold. Too often they're a self-congratulating pat-on-the-back and a detriment to success.
Please, prove me wrong. I know you can, because this does NOT apply to everyone.
Thanks for the comprehensive answer. I have similar experiences, treating the coverage numbers as a KPI will eventually trigger difficult decisions in writing tests.
What I’m wondering is what would make „more sense“ if one wants to track the code quality and test coverage in a broader sense.
* How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build?
* Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases).
* Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures.
Also, the flip side of testing is always monitoring.
* Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things.
* Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution.
* Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.
The overall test coverage of the code I'm currently unit testing is 63%. This does not mean the remaining 37% will ever get tested, it means around 63% of the codebase is worth testing. What's more relevant is the individual namespaces. For example: Services is at 100% test coverage, because everything in Services actually does something worth testing. Models.DTO, that's more like 50%. I do not need to test every auto getter and auto setter.
I view test coverage like a log curve, more is better, but the returns can get very diminishing.
As far as the tool, working in C# I use JetBrains dotCover.
Not directly, although I need to provide the tests themselves since I work in a regulated industry.
As others have mentioned test coverage is many times meaningless or misleading, can lead to abuse and don't contribute a lot to the overall quality of the product.
11 comments
[ 5.0 ms ] story [ 38.3 ms ] threadSee: https://github.com/jalal246/dflex/pull/158
What you end up with is:
1. Tests that at least invoke code that may be overlooked, but due to mocks, invoke it in a way thats meaningless.
2. Twice as much code to update when doing any feature work, refactoring, ... or even bug fixing!
3. A false sense of security because "tests passed!"
4. A lot of wasted time writing tests for code you end up either throwing out or significantly refactoring before it ever becomes a remotely active part of the project.
I'm jaded.
I LOVE good tests, but far, far too often they get in the way of real progress that actually matters to an end user. Tests are supposed to be worth their weight in gold. Too often they're a self-congratulating pat-on-the-back and a detriment to success.
Please, prove me wrong. I know you can, because this does NOT apply to everyone.
What I’m wondering is what would make „more sense“ if one wants to track the code quality and test coverage in a broader sense.
* How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build?
* Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases).
* Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures.
Also, the flip side of testing is always monitoring.
* Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things.
* Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution.
* Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.
The overall test coverage of the code I'm currently unit testing is 63%. This does not mean the remaining 37% will ever get tested, it means around 63% of the codebase is worth testing. What's more relevant is the individual namespaces. For example: Services is at 100% test coverage, because everything in Services actually does something worth testing. Models.DTO, that's more like 50%. I do not need to test every auto getter and auto setter.
I view test coverage like a log curve, more is better, but the returns can get very diminishing.
As far as the tool, working in C# I use JetBrains dotCover.
As others have mentioned test coverage is many times meaningless or misleading, can lead to abuse and don't contribute a lot to the overall quality of the product.
I use SonarQube/SonarCloud for the reports.