Does anyone else here dislike automated unit testing?
Where I work, we have unit tests that are all hand-coded. Even with that, it takes half an hour to run through the full suite - too long to do on any minor change, but still manageable when preparing a sandbox for checkin.
If we added in automated tests, we would have many times the number of tests, with a corresponding increase in the time taken to make them run. If you can't run unit tests quickly, their value is reduced.
I personally feel that a well-designed test framework should test all of the common use cases, some obvious edge cases, and should be easily adaptable so that when bugs are found, you can readily create a unit test that reproduces the bug, preventing regression.
I'm curious as to how others here decide which unit tests to write.
Sounds like you've got great test coverage, but the length of time it takes to run them is diluting their value, making you reluctant to run them as often and concerned about adding more.
Maybe you could offload/parallelize the test runs -- eg to a on-demand virtual hosting -- so as to minimize test run time as an issue?
This is probably the direction that we need to be taking. Another requirement that we have is that the code compiles without warnings for about 20 different target platforms. This takes hours, and we are currently writing a tool to parallelise those builds - we're hoping for a 20x speed increase. Maybe a similar approach could be achieved for running tests...
How about splitting your tests into a fast smoke test and a long test suite? You could run the smoke test before checking in. You could set up a continuous build system that would build with your latest changes, launch the long-running test suite, and email you the results.
In practice we run a subset of tests during development, but the current rule is that we should never checkin code that hasn't passed ALL of the unit tests. The idea being that each person is responsable for their own modifications passing all tests. If you check-in, and then get shifted onto another high priority task, and a later run of the unit tests discovers that a funky side effect from your changes breaks the unit tests, you are no longer available to fix the problem. Someone else has to do it, allowing you to escape responsability for your error. At least, that's the theory management here ascribes to...
I'd rather spend the effort generating code (UPDATE: and I do mean generating, not writing) than writing tests (presumably I could be generating tests at the same time). But there are enough people on the TDD train that there must be something of value there. Plus it's politically incorrect to say you don't like testing your code.
When people claim to be dispelling myths, I'd really wish they would do so with fact/numbers (ever see myth busters, they don't just dispel by telling you). I'd really like to see some numbers from the TDD folks on number of bugs found via TDD and it's effectiveness. And I don't want the story about the one bug found that saved them (just like that friend that lived because they were ejected from their car in an accident (fortunately they weren't wearing their seat belt)).
I've seen a lot of programmers go from writing no tests at all, to TDD (and pair programming, etc.). This extreme shift in behavior makes it seem more like a fad; I prefer a balanced approach. Use what works for you and your team, and if it stops working, do something else.
Has anyone here ever been in a situation where they thought, "Wow, a unit test would have saved me" or "Man, thank god for our unit tests?"
I don't think I've personally encountered that. Plenty of bugs at compile time or during regressions, but nothing where I could even imagine a unit test saving me.
I think the true value of unit tests is documentation. Instead of writing documentation that says, "Hey, this function should do this", you're writing code to do that, which is usually a pretty good idea. Executable documentation is the way to go. It's a very lisp-y idea.
That said, I'm not fond of the author's implication that unit testing will save you from cascading failures and make you a better programmer. He even says it's not a silver bullet, only to turn around and say it's the most magical thing to happen to software productivity since the invention of the keyboard.
It seems to me that the productivity gains are from forcing your developers to actually understand the code they're writing and interfacing with, but personally, I don't think I'd trust or hire someone who wasn't proud enough of their work to do that on their own.
9 comments
[ 5.0 ms ] story [ 237 ms ] threadWhere I work, we have unit tests that are all hand-coded. Even with that, it takes half an hour to run through the full suite - too long to do on any minor change, but still manageable when preparing a sandbox for checkin.
If we added in automated tests, we would have many times the number of tests, with a corresponding increase in the time taken to make them run. If you can't run unit tests quickly, their value is reduced.
I personally feel that a well-designed test framework should test all of the common use cases, some obvious edge cases, and should be easily adaptable so that when bugs are found, you can readily create a unit test that reproduces the bug, preventing regression.
I'm curious as to how others here decide which unit tests to write.
Maybe you could offload/parallelize the test runs -- eg to a on-demand virtual hosting -- so as to minimize test run time as an issue?
I wish the author would elaborate on that statement. I want supporting evidence, even if it's anecdotal.
When people claim to be dispelling myths, I'd really wish they would do so with fact/numbers (ever see myth busters, they don't just dispel by telling you). I'd really like to see some numbers from the TDD folks on number of bugs found via TDD and it's effectiveness. And I don't want the story about the one bug found that saved them (just like that friend that lived because they were ejected from their car in an accident (fortunately they weren't wearing their seat belt)).
I don't think I've personally encountered that. Plenty of bugs at compile time or during regressions, but nothing where I could even imagine a unit test saving me.
I think the true value of unit tests is documentation. Instead of writing documentation that says, "Hey, this function should do this", you're writing code to do that, which is usually a pretty good idea. Executable documentation is the way to go. It's a very lisp-y idea.
That said, I'm not fond of the author's implication that unit testing will save you from cascading failures and make you a better programmer. He even says it's not a silver bullet, only to turn around and say it's the most magical thing to happen to software productivity since the invention of the keyboard.
It seems to me that the productivity gains are from forcing your developers to actually understand the code they're writing and interfacing with, but personally, I don't think I'd trust or hire someone who wasn't proud enough of their work to do that on their own.