A nice article, and I think I should add the book to my (already too large) to-read pile.
I totally agree with "DRY might not be great in all cases" - In tests, it's often better to make each single test as expressive as possible. I once read (and forgot who said it):
"Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp)"
And here's a shameless plug: I recently wrote an article about how tests can help you achieve simple design, and the next article in the series will be coming soon: http://quickglance.at/en/simple_design/passes_its_tests
> "Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp)
I'm getting so tired of people who try to capture the intricacies and complexities of our profession with witty one liners that have to be taken like dogma.
Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates.
Who says that they have to be taken like dogma? Sometimes the one-liners are just easy to remember. Often they are funny. They can be a great tool to quickly illustrate a point or start a conversation.
The one liners can never replace a conversation or explanation. They cannot replace common sense or real knowlege. I would never treat them as laws or follow them by the letter...
Sadly that's with post-processing improvements. Without the guests recording their own audio and then syncing it after (which is something of a big ask), we're reliant on the quality of the connection. Something to work on for sure. I'm hopeful solutions like http://podclear.com can help.
Yeah, sorry about that. Derrick has one, but was moving house at the time of recording and it was packed away. So future ones should be using that, at least on the host side.
This is one of the most sensible articles I have seen on unit testing. Rather than religiously insist on tests first or 100% coverage, judge when it is valuable to have these things and when they are not important. Far too many articles on testing test the framework / language being used.
Terrible audio. Next time you interview someone, ask them to install Audacity and to run it during the interview. Then have them send you the audio file via Google Drive.
I constantly have problems with people posting the same information on unit tests, again and again. I know how to set up (for example) RSpec, MiniTest, Cucumber, FactoryGirl. The thing that people struggle with is how to use these in a larger, practical sense, rather than in just the trivial. I think this puts a lot of people off testing.
Agreed. I have only used TDD once as a front-end web developer, and I've found that I do struggle with the practicality of it. I spent half my time managing and maintaining my tests. When I did the math, It seemed that all that extra time spent was more than It would have taken to simply chase down bugs that would have arisen in a non-tdd environment. It just seems to take more time than is practical. Especially in agile startups.
Perhaps. The difference is that in one case you'll have the machine automatically tell you when and why your code stops working because of a regression, and in the other you have no clue until the customer hits it. It's not only about proving it worked once, it's about proving it with every commit.
Read that and read oldie, but goodie Extreme Programming Explained, and IMO they do a good job of painting the bigger picture.
TL;DR Tests are live specs that break when you change the underlying spec of your code without updating the spec in the test. If you can express your code's spec in a test, then you have a much higher chance of deeply understanding your design... hence the article's recommendation that you first know what you are doing before diving into writing tests.
As for tests in the larger practical sense vs. the trivial... The vast majority of large systems are compositions of small, seemingly trivial pieces. A large system's code needs to be organized so that all its component pieces/concerns are comprehensible and separate; otherwise, it's an unmaintainable monolithic soup. The unit tests adhere to the same principle, in general, and follow the composition of the system.
I like Jay's comments about how tests are wasted if you don't know exactly what you want, and just try out different things (because then you need to change/delete the tests when you end up changing your code).
As for common mistakes with unit tests, my top 5 list is:
Here is another one: failing to understand that tests introduce additional maintenance and design overhead.
Quis custodiet ipsos custodes?
Tests are additional code that is tightly coupled to the target code that is meant to be used in production. This additional code requires design, debugging, and maintenance just like the target code. It is also tightly coupled to the target codebase: a change in either necessitates at least an inspection to determine what, if any, changes must then be made in the other.
This all adds overhead to the whole process, and too often I see casual, glib statements implying or even asserting that the overhead is nonexistent or else pays for itself. It isn't free and it is especially not free the more dogmatic the test proponents and tests themselves are.
Agreed. I think Jay touches on this when talking about how you don't need to aim for 100% unit test coverage.
It's also what I mean by testing algorithmic code (where the pay off from the effort is large), versus testing coordinator code, where you are better off not unit testing it (because it is not complicated).
Code that glues together and orchestrate interactions between other code units etc. It's hard to test because of all the dependencies, but the actions it performs are not very complicated, so not worth the effort of unit testing.
Your complaints are actually the whole reason tests exist. Of course you know that your code works when you write it. You then write tests that are tightly coupled to the behavior that you require from the code to run it through its paces as you work on it.
Speaking of coupling, tests prevent tight coupling between different pieces of your codebase. Since there are then at least 2 consumers of any piece of code, the implementation does not get coupled to a particular one as easily.
Also, if you would like to work with a module or microservice based approach, tests are simply necessary to work. If you're storing different pieces of the codebase in different repos, you will need to have tests to work on them at all.
Basically, if you are "testing" your production code with other pieces of your production code (I hope you don't do this), it will become tightly coupled. Test your code with tests.
I'm not sure we disagree about the purpose and utility of tests. I certainly was not complaining about tests. If anything, I was complaining that too often, in my experience at least, it seems to be presented as an undeniable truism that, generically, 'tests' are always a benefit, and usually the more tests the better.
My point is that tests (at least, unit/integration tests of the variety usually mentioned in TDD) are more code that has to be maintained (and designed, and debugged). They incur overhead and as such impinge on development resources. As a result tests introduce the need for considering tradeoffs between developing tests and developing the software product itself. Simply writing tests for the sake of having tests to test code is naive; at least as naive as assuming that once working a given piece of code works forever, but it is a mistake that I've witnessed on more than one occasion.
Testing obviously takes time. The problem with not having tests is that it takes time, but you don't necessarily know that it's taking time or how much. Lots of people think they are too smart to need tests, and it's just a waste of their time. But they are spending huge amounts of time fighting bugs, and even more bugs are getting out to users.
I think what they are saying is the kind of tests they see in a corporate environment are not the kind that add any value to the refactoring process (or worse, inhibit it).
Tests are a great thing when everyone involved understands their value. In my former team we had a big number of unit & integration tests that saved my ass many times.
On the other hand, when the team has a KPI "we need a big test coverage!" and people are lazy or simply bad developers, you end up with a code like
Rant about "unity". The idea of unity is vague. Almost every person has a different definition of unity. Do not think about unity think about granularity, or even better facets of behavior. The amount of integration can scale to infinity. You can have infinity layers of integration between unity and end2end tests. The attempt to classify granularity is a waste of time.
I write lots of unit tests. One thing I'm struggling with now is the organization of a large number of tests. It's hard to know what I have tests for without going into my tests directory and reading comments. Even then, I feel like I could do more to have a "big picture" overview of my testing strategy rather than seeing a bunch of one-off tests. I haven't quite figured out what that should look like.
I think the IDE could do more in some cases. For example, when unit testing a method, I'd like to be able to tag a reference to the method I'm testing. Then, when I'm in my code, my IDE could show me the docstrings of tests I have for that method.
It's the difference between "hello world" and software design principles used to build large scale systems. I see lots of "hello world" for testing, but not much of the other side. Does anyone have suggestions?
The way I personally solve this problem is by organizing my code and my tests in a similar structure. Split application code up into reasonable files, and split the tests up the same way. Depending on your language/test framework, it also helps to put the test right next to the code... i.e. in Go, you can have component.go and component_test.go in the same directory.
I do agree though that it would be nice to see an IDE have some smarts about tests...
Visual Studio + Resharper lets you do a "Find Usages" of a type or method which is very useful for finding relevant tests. Intellij has the same functionality and I imagine the other JetBrains IDEs are similar.
More general advice is that your test structure should mirror your code structure. So a folder called Services should have a matching test folder called Services containing tests for the classes within it.
I have to say that the trade-off argument brought up between DRY and readability is pretty lame. If you have a junior developer who can't make sense of the test because you put the test data in a separate class or are using a setup method, then teach him or her how to trace the call to the setup method and then read through it.
If we follow this logic then all code should follow the same methodology of readability because tracing through method calls is too cumbersome. Obviously that isn't the case for most code, so why would it be true for unit tests?
I don't think having readable tests and DRY are necessarily mutually-exclusive. If you name methods and variables with lengthy but very descriptive names your tests can still be readable. In fact the more you modularize your setup methods the more readable they can become. For example:
...is pretty readable despite being broken up. It may or may not be true that if you just went through the setup here it may be more readable, but the benefit to this approach is that you only have to write a method to add an SSN to a person object one time and then everyone can use it without needing to understand the structure of different objects. Notwithstanding readability, I would argue that DRY makes writing tests easier than cramming a bunch of setters into every test case. Once somebody understands what a setup method does, in every place that it is used in other tests the tester can skip over that part and move on to testing their specific use case.
Does anyone have any resources for best practices unit testing web apps written with frameworks like Django, Rails, etc? Whenever I try to apply TDD to a framework, I find myself testing the functionality of the framework (which should already be tested), rather than my app itself. How do you decide what to test when you are effectively cobbling together framework concepts? Is this the time for focus on integration tests over unit tests?
43 comments
[ 4.3 ms ] story [ 45.2 ms ] threadI totally agree with "DRY might not be great in all cases" - In tests, it's often better to make each single test as expressive as possible. I once read (and forgot who said it):
"Code should be either DRY (don't repeat yourself) or WET (write expressive tests), but never damp)"
And here's a shameless plug: I recently wrote an article about how tests can help you achieve simple design, and the next article in the series will be coming soon: http://quickglance.at/en/simple_design/passes_its_tests
I'm getting so tired of people who try to capture the intricacies and complexities of our profession with witty one liners that have to be taken like dogma.
Just use your common sense, let your experience guide you and if you're not sure about something, ask your teammates.
The one liners can never replace a conversation or explanation. They cannot replace common sense or real knowlege. I would never treat them as laws or follow them by the letter...
Brilliant, added to our bugzilla quips list.
Hopefully this will become a mainstream thought. Then maybe we will fix the legacy browser languages problem. Man can dream...
Read that and read oldie, but goodie Extreme Programming Explained, and IMO they do a good job of painting the bigger picture.
TL;DR Tests are live specs that break when you change the underlying spec of your code without updating the spec in the test. If you can express your code's spec in a test, then you have a much higher chance of deeply understanding your design... hence the article's recommendation that you first know what you are doing before diving into writing tests.
As for tests in the larger practical sense vs. the trivial... The vast majority of large systems are compositions of small, seemingly trivial pieces. A large system's code needs to be organized so that all its component pieces/concerns are comprehensible and separate; otherwise, it's an unmaintainable monolithic soup. The unit tests adhere to the same principle, in general, and follow the composition of the system.
As for common mistakes with unit tests, my top 5 list is:
1. Testing algorithms together with coordinators.
2. Mocking too much.
3. Not using asserts.
4. Leaving print statements in the tests.
5. Checking the log statements, not the result.
More details here: 5 Unit Testing Mistakes http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/
Quis custodiet ipsos custodes?
Tests are additional code that is tightly coupled to the target code that is meant to be used in production. This additional code requires design, debugging, and maintenance just like the target code. It is also tightly coupled to the target codebase: a change in either necessitates at least an inspection to determine what, if any, changes must then be made in the other.
This all adds overhead to the whole process, and too often I see casual, glib statements implying or even asserting that the overhead is nonexistent or else pays for itself. It isn't free and it is especially not free the more dogmatic the test proponents and tests themselves are.
It's also what I mean by testing algorithmic code (where the pay off from the effort is large), versus testing coordinator code, where you are better off not unit testing it (because it is not complicated).
For more, see: http://blog.stevensanderson.com/2009/11/04/selective-unit-te...
Speaking of coupling, tests prevent tight coupling between different pieces of your codebase. Since there are then at least 2 consumers of any piece of code, the implementation does not get coupled to a particular one as easily.
Also, if you would like to work with a module or microservice based approach, tests are simply necessary to work. If you're storing different pieces of the codebase in different repos, you will need to have tests to work on them at all.
Basically, if you are "testing" your production code with other pieces of your production code (I hope you don't do this), it will become tightly coupled. Test your code with tests.
My point is that tests (at least, unit/integration tests of the variety usually mentioned in TDD) are more code that has to be maintained (and designed, and debugged). They incur overhead and as such impinge on development resources. As a result tests introduce the need for considering tradeoffs between developing tests and developing the software product itself. Simply writing tests for the sake of having tests to test code is naive; at least as naive as assuming that once working a given piece of code works forever, but it is a mistake that I've witnessed on more than one occasion.
Refactor code, and it gives you that much more confidence you have not broken anything.
They're seriously useful mid-later stages of a project, which is probably why startups devs aren't doing them...
On the other hand, when the team has a KPI "we need a big test coverage!" and people are lazy or simply bad developers, you end up with a code like
Well actually if the organization is that dysfunctional it's probably game over, but Mutation Testing can catch you when you do this accidentally.
I think the IDE could do more in some cases. For example, when unit testing a method, I'd like to be able to tag a reference to the method I'm testing. Then, when I'm in my code, my IDE could show me the docstrings of tests I have for that method.
It's the difference between "hello world" and software design principles used to build large scale systems. I see lots of "hello world" for testing, but not much of the other side. Does anyone have suggestions?
I do agree though that it would be nice to see an IDE have some smarts about tests...
More general advice is that your test structure should mirror your code structure. So a folder called Services should have a matching test folder called Services containing tests for the classes within it.
If we follow this logic then all code should follow the same methodology of readability because tracing through method calls is too cumbersome. Obviously that isn't the case for most code, so why would it be true for unit tests?
TestData.CreatePerson();
TestData.AddSsnToPerson();
TestData.PersonModule.checkWhetherPersonHasValidSsn();
...is pretty readable despite being broken up. It may or may not be true that if you just went through the setup here it may be more readable, but the benefit to this approach is that you only have to write a method to add an SSN to a person object one time and then everyone can use it without needing to understand the structure of different objects. Notwithstanding readability, I would argue that DRY makes writing tests easier than cramming a bunch of setters into every test case. Once somebody understands what a setup method does, in every place that it is used in other tests the tester can skip over that part and move on to testing their specific use case.