Ok, TDD didn't help with designing an algorithm for Lychrel number - because you just don't get around implementing the loop that tests if a number is a lychrel number, and it's very simple.
But to me that's not the whole purpose of unit tests - the most important part for me is that it tells you very quickly when you have bugs in your implementation. Somehow the author neglects to address this very central point.
"This is surely the most ignominious debacle of a project listed on my site, even though others have also not shipped. (Sudoku did not ship and will not. Shotgun will go forward if the Customer wants to.)"
Listen kids, testing is a tool for HELPING THE DEVELOPER, not for using to engage in a "more pious than thou" dick-swinging my Cucumber is bigger than yours idiocy.
Testing is about giving YOU THE DEVELOPER useful and quick feedback about if you're on the right path, and if you've broken something, and for warning people who come after you if they've broken something. It's not an arcane methodology that somehow has some magical "making your code better" side-effect...
The whole concept of "test driven development" is hocus, and I speak this as someone who writes a lot of tests, and who charges a lot of money for fixing test suites. Instead: developer-driven testing. Give your developers useful tools for solving problems and supporting themselves, rather than disappearing in to some testing hell where you're doing it a certain way because you're supposed to.
It's hocus where contrasted to developer-driven testing, because Test Driven Development - as a development methodology (as opposed to a tool) - espouses that you Write All Your Tests first.
Have I had experience (and much value) out of sometimes writing tests for certain problem classes before writing any code? Yes. Changes to existing functionality are often a good candidate.
Does TDD as a methodology suggest you should ALWAYS write your tests first? Everything I've read suggests so.
And "as any fule kno" [sic], this is idioicy during a design or hacking or greenfield phase of development. Allowing your tests to dictate your code (rather than influence the design of modular code) and to dictate your design because you wrote over-invasive test.
tl;dr: Writing tests before code works pretty well in some situations. Test Driven Development, as handed down to us mortals by Agile Testing Experts and other assorted shills, is hocus.
This is a common misconception. TDD does not 'espouse' in any way that you write all your tests first. Nothing I have read suggests so.
For me, TDD, in practice means switching back and forth between writing tests and production code. That is, write some test code, run, fail, fix the production code, write some more test code. It is incremental.
> Writing tests before code works pretty well in some situations. Test Driven Development, as handed down to us mortals by Agile Testing Experts and other assorted shills, is hocus.
What? So it works in some situations, but not all. Thus hocus?
I'm no 'Agile Testing Expert', but I have worked for the last 2-3 years doing TDD exclusively (as in ALWAYS writing tests first). It isn't hocus. It is effective. I can understand it doesn't suit all problems, and can potentially be a hinderance sometimes, but that is the same for ANY methodology/practice. I don't quite understand your hostility towards it.
Excuse me for interrupting the budding flame war, but I have a question regarding the subject, which you seem qualified to answer? :-)
My experience is the same as (what I believe is) the usual criticism of tests-first.
I often need to rethink the API to my functions/methods, sometimes more than once. This goes for both external and internal APIs, so it doesn't help with detailed interface specifications between modules.
Test-first is a problem, since it needs a fixed interface before I'm ready to lock down the interface.
I tend to instead write inside-out; a few pages of code and then write tests for that code. That model isn't really kosher/halal/etc, according to TDD?
Interesting TDD argument. I'll buy a book or something and try again when my present hobby project is done and I have time for experimenting.
(IDE refactoring needs a stricter language than I prefer. Also, without an IDE and using my old eyes I have 2 A4 of code + 1 A4 of bash on a 24" monitor...)
If you're looking for book suggestions, I'd recommend "Growing Object-Oriented Software, Guided By Tests" by Steve Freeman and Nat Pryce. It's pretty pragmatic, and the treatment of the subject is pretty thought-provoking (at least, I thought so when I read it).
According to TDD writing inside-out is one of the points it tries to combat.
The point of using TDD to write interfaces is that you first write the code you wish you had, and then implement it. The other way around you could end up with code that is optimized for the code inside the module, and not optimized for using it.
Of course now you have to rewrite your tests and your code when changing an API, but at least you know that no other part of your application got broken by the new API.
I personally like the TDD approach to write interfaces, since then I won't take shortcuts just because it is easier to do in the specific module that I'm programming/extending. It will actually produce code that can be used easily from the outside and removes a lot of dependencies in the process.
> I can understand it doesn't suit all problems, and can potentially be a hinderance sometimes
Then why do it exclusively?
Testing is about helping developers, and recognizing that automated testing is about benefit to developers, rather than cargo-culting a workflow and decreeing that one size fits all.
Writing tests first as a tool to be deployed where it works is "Developer Driven Testing" - focussing on making the developer more productive by choosing the right tool for the job. Generalizing a bunch of testing rules and saying This Is The One True Way Even When It Isn't (I mean seriously, you continued to use it when it was a hindrance?!) - that's not right.
Primarily I do web development, with python and ruby, TDDing is VERY well supported, and thus has allowed me to do it exclusively. But I can imagine there are systems out there that are difficult to TDD, web apps are not among them though.
Sounds like you are interpreting TDD in a rather extreme way. If you haven't found it useful, then fine don't use it. But that doesn't mean it is universally bad, and thus doesn't deserve the "It's all hocus and you are a stupid zealot if you do it"
I don't think I've ever seen a description of TDD that suggests you write all of your tests before writing any production code - rather you iteratively write some tests, have them fail then write the code to make them work and then see the tests pass, repeating that process until you have implemented everything you need.
Sure, but you write /some/ tests first. And that means, if you want to do it meaningfully, that you either need to zoom down in to tiny bits of functionality first in order to be able to write those tests (see the isPalindrome discussion in the article), or you write a test that requires most of the software to be finished, or you cheat and fudge it.
Making tests a central part of the process because they're useful to developers? Awesome. Dictating a workflow to developers that works in some cases as the One True Way: ridiculous.
I've always thought of testing as simply a way to verify that your code does what is supposed to do.
I tend to agree with you that TDD as a development methodology is deeply flawed. I have never found that the act of writing a test case and making it pass has helped me with the actual creative process of solving a problem via code. In fact, I'll frequently dash out a method I'm writing quickly, and then go back and reconstruct it piece by piece via TDD.
The main benefit writing tests first gives you is that you know your test is actually working (i.e. because it first fails) and that the code you implement actually passes it. It also keeps you honest because it prevents you from writing "extraneous" code, i.e. code that exists but is not specified by a test.
So yeah, I totally agree with you that the cult of TDD kind of sucks. But I haven't stumbled upon a better way yet. I've often thought it would be cool if there was some way for e.g. your version control software to be tied to the code you write that passes certain test(s), such that you could somehow verify that without that code your test fails and with that code your test passes. For the present though, TDD tends to work pretty well for me.
I think you're narrowing in on the distinction I'm trying to make between a methodology and a tool there; writing tests before you implement a piece of functionality is a useful tool to have in your testing toolbox.
Labouring under the idea that Tests Must Come First (and everything I've seen, and everything I /do/ see now suggests that that is the central idea in TDD - you write a test, then you write the code to pass it) without pivoting to see that testing is a useful practice in so much as it helps developers is the wrong approach.
I think you've hit the nail on the head, @peteretep.
The problem is that ideologies like TDD try to come up with simple rules for how to develop (i.e. always write tests first). In reality, it's better to use your brain when you're deciding how to approach something (i.e. is this a good situation for writing tests first?)
Perhaps by permuting your code the system could determine if there was anything that wasn't required to make the test pass and exclude it from the check-in.
I often code three or four commits worth at once when on a roll and it's a pain doing multiple staged adds, stashes, tests, and commits to store each sub-change on its own. Having a tool to help with this would be great.
> And "as any fule kno" [sic], this is idioicy during a design or hacking or greenfield phase of development.
When you write something you run it to make sure it works. Bottle that and call it a test. And then merely do this before you write the code. "What will I be done when I get this code to do?"
Don't plan, and spec, your whole API before you've written a line of code or anything, but do write a one-liner to check whatever it is you're going to do so you know when you're done.
Ideally you'll write a test (still just a one-liner) for every corner-case that you cover in code, because you need a failing test to know if your fix actually worked.
And face it, if you're doing "real" development work you've got business goals, and a spec with features intended to deliver on that, and you need to cover every tiny detail. If you don't have the spec ('fail on values over n') directly linked to the code that implements it, it will get lost, and if you aren't testing it you can't legitimately claim to be delivering it.
A spec, such as you'd get by performing useful TDD, is a requirement for modern software development. You could write it all up-front in the planning phase (waterfall style), or all after in the testing phase (waterfall style again), or you could interleave it with research and development, which if you think about it, sounds kind of agile by comparison...
The issue/argument here is not that writing tests is worthless but that in many cases the approach of "if we write enough tests then the code that solves the problem will eventually emerge" is doomed to failure or at least results in an extremely verbose approach to development.
But then I think the approach is the wrong one. In TDD you are not required to write a lot of test and hope that they magically work in the future. Instead a small test to do one thing should be written, and then write enough code to make it pass.
For example lets take a simple newsletter signup form with only a field to enter an email address.
Now you shouldn't write a test to see what happens when you enter an email, a wrong email, if the confirmation email is sent, etc. Start with the simplest case: "If I fill in my email, and I press klick, then it should show me a confirmation that I singed up." Not more. Now make that pass, and you have working code!
Now you can add tests that verify that a wrong or empty email is rejected, now make that pass. More working code!
With TDD you should never end up with a lot of tests and no code that solves the problem. In that case you are actually writing too many tests and not enough code.
Which we demonstrably do as an industry. When neither NASA nor Knuth can write 100% correct code, the rest of us can't reasonably expect to pull it off. Human beings doing meticulous work make lots and lots of mistakes, and I welcome any feasible tactic that stops mine from being released. Sure, automating "I got a confirmation after pushing the button" seems hokey on day one (it's not as if you'd otherwise commit and go home before having seen it work) but somebody is eventually going to make a change that unexpectedly breaks it, and I want to know about it.
I'm not debating the importance of tests, but rather of the test-driven approach. I can write code and then write tests (or intermingled) and satisfy your point. I don't have to switch my brain into this wierd test-driven approach to do that. TDD zealots don't seem to understand that.
People just assume older code works. I'm regularly finding stuff we've had in production for months that never did what it was supposed to. You could go back and write all the same tests after the code is done, but I find it extremely unlikely that anyone will actually reach the same degree of test coverage (I know I for one never have). And even if you're one of the rare few who will pay down this technical debt eventually, it takes away quite a bit of the value of the tests to muddle through all the coding work without them.
I'm seeing a lot of misconceptions about test-driven development in this thread. The biggest one is that test-driven development involves writing a bunch of tests first. Instead, TDD involves interweaving tests and production code in a very tight cycle. You write a few lines of test code, then a few lines of production code, then you review and improve your code, then write a few lines of test code, and so forth. Each step only takes a minute or so.
This is useful because you're constantly making and checking hypotheses: "This code should make a test fail in X way... yup. Okay, now this code should make the test pass. Yup. Now this change shouldn't change behavior, and the tests should still pass. Oops! Undo. Oh, that method has to be called in the constructor. All right, this change should work. Good."
Despite the extremists, TDD really is a useful tool--and that's all it is. Sure, TDD has been turned into yet another vehicle for programmers to engage in religious wars, but you can't really blame TDD for that, any more than you can blame Richard Stallman for emacs vs. vi wars.
There are a couple tricks to effective unit testing. Ultimately though, the goal is to write non-brittle tests. That is, tests that won't break due to unrelated changes. Get this wrong and the cost of maintaining your tests will outweigh the benefits.
Achieving this, in my experience, comes down to disciplined used of mocks and stubs. I've seen people lean too heavily on them, or not heavily enough. The biggest problem I've seen, particularly common in the Java world due to jMock's datedness, is over specifying expectations on mocks. Stubs which can provide canned responses (whether through a framework or manually rolled) are absolutely and totally the most underused yet useful tool in your testing arsenal.
Can you expand a bit on the distinction you're making between mocks and stubs, and why you believe stubs are both underused and useful?
Perhaps I'm doing it wrong, but I've found in my experience that bugs I miss in testing increase proportionally with my use of mocks or stubs. Invariably, when I'm forced to stub something and replicate its behavior I miss something subtle that comes back to bite me.
First, people rely too heavily on mocks/stubs/fakes/(whatever you want to call them). This has gotten better in the past 4 or so years (in my mind, largely because of Rails and it being "acceptable" to hit a DB in a test - though it might actually predate Rails). I think this problem is pretty straightforward to understand (again, especially when you look at the Rails way to test a DB interaction versus a more "traditional" way).
However, there are instances where mocks/fakes/stubs are important. Some outside dependencies might not be accessible during testing, might not be predictable/deterministic, might be too slow or might simply involve too much setup. Also, solely relying on "real" implementations can also make your tests too brittle. Why should the logic for can_legally_drink? break if the DB column is changed from DOB to dateofbirth (ok, that's an extreme and poor example, but you get the idea, hopefully).
Anyways, assuming you agree that sometimes a fake is simply better, you get into mocks vs stubs. To me a stub is dumb and a mock is strict. Stubs also automatically reply with canned answers, mocks don't. A mock is used to assert that a certain expected call was made. A stub is used just to get your test to move over a line of code.
The problem is that people use mocks over and over again. Re-specifying the same expectations in every test..making X tests break when you change the behavior of the interaction, versus just 1 test. As semi-pseudocode (and yes, it might be better to just hit the DB in this case):
def self.login_user(name, password)
user = Store.get_user_by_name(name)
return user.nil? || user.password != password ? nil : user
end
To me, you kinda wanna check that the above code properly interacts with someStore. This is when you use a mock:
it "gets the user from the store" do
Store.should_receive(:get_user_by_name).with('leto')
User.login_user ('leto', 'ghanima')
end
You also want to make sure the password matching works. This is where people go wrong. They'll use a mock (strict) again..which'll just repeat the above code...except this test has really nothing to do with how store...we just want to get over the line of code:
it "returns nil if the passwords don't match" do
Store.stub!(:get_user_by_name).and_return(User.new)
User.login_user('leto', 'ghanima').should be_nil
end
If you are familiar with jMock, it's kinda the difference between allowing and oneOf. oneOf is very strict, allowing isn't. You really should use allowing whenever you aren't explicitly testing the interaction (and you shouldn't explicitly test the interaction in more than 1 test (dedicated to testing said interaction)).
Some framework even return smart canned answers. So instead of returning nil, they'll default to a new instance of the type, or an empty array, or some default scalar value (like an empty string). This is particularly useful when you just want to get passed a null reference (which is exceedingly common).
Thanks, it makes perfect sense and is pretty much why I virtually never use mocks. In fact, my personal philosophy is at odds with most advice I have seen around unit testing, in that I try to avoid the use of even stubs wherever possible. In other words, I try to make my unit tests as much like integration tests as possible. Obvious exceptions are things like third-party APIs, which must be stubbed, but everything else I try to get close to production code.
I've had similar experience with strict mocks. However, mocks don't need to be strict. In Java, for example, the current mocking tool of choice is Mockito, which returns "nice" mocks that return nulls, empty collections and such.
I can't seem to find a blog post discussing the merits, but IME strict mocking leads to a lot of tests breaking just because you added one more piece of functionality to a method. Now, this causes you to edit all sorts of existing tests to cause them to ignore a call to collaborator. While you can extract common expectations and such to eliminate duplication, you still end up with many tests failing due to one small change in behavior.
Normally your tests tend to assert one or two behaviors per test. With strict mocking, your tests become awkwardly cluttered by setting up expectations/interactions for things that are unrelated to what they are asserting.
With nice mocking, it's much easier to get away with only the expectations/interactions that are directly relevant to what a single test is asserting. This also means that you get more fine-grained failures, which speeds up diagnostics.
That's an interesting question - would a unit test prevent this error?
To answer this, let's consider what happened: the database didn't respond in an appropriate way, and probably returned an error. I'm not exactly sure what exact error was displayed, but WordPress has caught this situation, and handled it the best it could - displaying a proper error.
I would write a unit test for this exact scenario - using a fake (mock) object, simulate the database timeout, and make sure that the error was caught and handled appropriately. Other then that there's no real other way to gracefully recover.
When writing code it is imperative to to get great feedback, as often as possible. In a number of cases, TDD helps you do that. In cases where it don't, well, it doesn't, so don't use it.
Maybe I'm in a mad mood or somth (it's Monday, after all) but all this "let's write a parallel world, called TDD, to which the actual production code has to comply and praise and give sacrifices"-mantra has gotten a little over the top.
Don't get me wrong, I'm sure that all these guys and chicks that live and die by TDD are smart (probably smarter than me), but what they are building is starting to get more and more out of touch with real-world requirements (I wanted to say somth about how this resembles a schizophrenic world, but for the moment I'll stop short of that).
My father was a civil engineer, at a time when they didn't have AutoCAD available and all that fancy computer stuff. As he elegantly put it: "if I mess things up in my work people will die, either in 2 weeks or in 20 years, when the next big earthquake will hit us". Well, I never saw him "testing" building apartment-blocks, or industrial buildings, or roads, he made sure that what was on the sheet of papers where the construction plans were drawn would get built as accurate as possible in real life. And keep in mind that he was managing construction workers, many of them close to illiterate, even former convicts, and not CS-graduates. That's the job of all civil engineers from all over the world. So, if they can do it, why in the name of God do we programmers write tests for mundane stuff like sudoku-solvers? Something is not right.
Civil Engineers these days (or this is my understanding, anyway) DO use AutoCAD and 'all the fancy computer stuff' because testing allows you to code more productively, when you do it right. That's why engineers these days do automated simulations...
At the point where you're writing tests to impress other programmers (see article) about how much of a 'craftsman' you are, you lose. When you're writing tests that help you as a developer, and help get the software out more quickly and with higher quality, then you're doing it right.
Surely he did though. At some point he (roughly) had to add up the weight of the building and make sure the supports were thick enough to bear it. It's just too basic to talk about.
Here's my trouble with testing as espoused by proponents of TDD: I bought into the Agile Manifesto. I believe in working code over specifications. How then, am I supposed to write a bunch of tests that will in turn tell me what code to write?
It is not a coincidence that one of the reasons TDD'ers love rspec so much is that the "code reads like documentation" - it's even called r spec.
Perhaps that is ok for mature projects. But I often hear TDD and agile used together as if they weren't in conflict and I get confused. Then again, I've only ever worked on new projects.
51 comments
[ 2.9 ms ] story [ 117 ms ] threadBut to me that's not the whole purpose of unit tests - the most important part for me is that it tells you very quickly when you have bugs in your implementation. Somehow the author neglects to address this very central point.
Ravi's article: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s...
Peter Norvig's solver: http://norvig.com/sudoku.html
Ron Jeffries' attempts:
http://xprogramming.com/articles/sudokumusings/
http://xprogramming.com/articles/oksudoku/
http://xprogramming.com/articles/sudoku2
http://xprogramming.com/articles/sudoku4
http://xprogramming.com/articles/sudoku5
And, as dessert (Ron is very frank about his failures):
http://xprogramming.com/articles/roroncemore/
"This is surely the most ignominious debacle of a project listed on my site, even though others have also not shipped. (Sudoku did not ship and will not. Shotgun will go forward if the Customer wants to.)"
Listen kids, testing is a tool for HELPING THE DEVELOPER, not for using to engage in a "more pious than thou" dick-swinging my Cucumber is bigger than yours idiocy.
Testing is about giving YOU THE DEVELOPER useful and quick feedback about if you're on the right path, and if you've broken something, and for warning people who come after you if they've broken something. It's not an arcane methodology that somehow has some magical "making your code better" side-effect...
The whole concept of "test driven development" is hocus, and I speak this as someone who writes a lot of tests, and who charges a lot of money for fixing test suites. Instead: developer-driven testing. Give your developers useful tools for solving problems and supporting themselves, rather than disappearing in to some testing hell where you're doing it a certain way because you're supposed to.
I've worked on numerous projects where TDD has (and continues) to prove invaluable.
So regardless how much experience you've got, I've had experience to the contrary, thus the "whole concept" isn't hocus.
Why have you come to believe this?
Have I had experience (and much value) out of sometimes writing tests for certain problem classes before writing any code? Yes. Changes to existing functionality are often a good candidate.
Does TDD as a methodology suggest you should ALWAYS write your tests first? Everything I've read suggests so.
And "as any fule kno" [sic], this is idioicy during a design or hacking or greenfield phase of development. Allowing your tests to dictate your code (rather than influence the design of modular code) and to dictate your design because you wrote over-invasive test.
tl;dr: Writing tests before code works pretty well in some situations. Test Driven Development, as handed down to us mortals by Agile Testing Experts and other assorted shills, is hocus.
For me, TDD, in practice means switching back and forth between writing tests and production code. That is, write some test code, run, fail, fix the production code, write some more test code. It is incremental.
EDIT: Fixed a typo
What? So it works in some situations, but not all. Thus hocus?
I'm no 'Agile Testing Expert', but I have worked for the last 2-3 years doing TDD exclusively (as in ALWAYS writing tests first). It isn't hocus. It is effective. I can understand it doesn't suit all problems, and can potentially be a hinderance sometimes, but that is the same for ANY methodology/practice. I don't quite understand your hostility towards it.
What do you mean by 'developer driven testing'?
The hocus is that you must use it, because some guru said so.
My experience is the same as (what I believe is) the usual criticism of tests-first.
I often need to rethink the API to my functions/methods, sometimes more than once. This goes for both external and internal APIs, so it doesn't help with detailed interface specifications between modules.
Test-first is a problem, since it needs a fixed interface before I'm ready to lock down the interface.
I tend to instead write inside-out; a few pages of code and then write tests for that code. That model isn't really kosher/halal/etc, according to TDD?
TDD suggests 3 steps: 1. Write one test 2. Write the minimal code to make it pass 3. Refactor the code you've got.
Try not to think of tests as immutable. You can throw away tests when they don't provide value.
Finally, make sure you're using an IDE with refactoring support. That helps immensely when you make an API change which will affect your tests.
Interesting TDD argument. I'll buy a book or something and try again when my present hobby project is done and I have time for experimenting.
(IDE refactoring needs a stricter language than I prefer. Also, without an IDE and using my old eyes I have 2 A4 of code + 1 A4 of bash on a 24" monitor...)
The examples are all Java, but the accompanying website (http://www.growing-object-oriented-software.com/) has links to reimplementations in several other languages.
The point of using TDD to write interfaces is that you first write the code you wish you had, and then implement it. The other way around you could end up with code that is optimized for the code inside the module, and not optimized for using it.
Of course now you have to rewrite your tests and your code when changing an API, but at least you know that no other part of your application got broken by the new API.
I personally like the TDD approach to write interfaces, since then I won't take shortcuts just because it is easier to do in the specific module that I'm programming/extending. It will actually produce code that can be used easily from the outside and removes a lot of dependencies in the process.
Then why do it exclusively?
Testing is about helping developers, and recognizing that automated testing is about benefit to developers, rather than cargo-culting a workflow and decreeing that one size fits all.
Writing tests first as a tool to be deployed where it works is "Developer Driven Testing" - focussing on making the developer more productive by choosing the right tool for the job. Generalizing a bunch of testing rules and saying This Is The One True Way Even When It Isn't (I mean seriously, you continued to use it when it was a hindrance?!) - that's not right.
Sounds like you are interpreting TDD in a rather extreme way. If you haven't found it useful, then fine don't use it. But that doesn't mean it is universally bad, and thus doesn't deserve the "It's all hocus and you are a stupid zealot if you do it"
Making tests a central part of the process because they're useful to developers? Awesome. Dictating a workflow to developers that works in some cases as the One True Way: ridiculous.
I tend to agree with you that TDD as a development methodology is deeply flawed. I have never found that the act of writing a test case and making it pass has helped me with the actual creative process of solving a problem via code. In fact, I'll frequently dash out a method I'm writing quickly, and then go back and reconstruct it piece by piece via TDD.
The main benefit writing tests first gives you is that you know your test is actually working (i.e. because it first fails) and that the code you implement actually passes it. It also keeps you honest because it prevents you from writing "extraneous" code, i.e. code that exists but is not specified by a test.
So yeah, I totally agree with you that the cult of TDD kind of sucks. But I haven't stumbled upon a better way yet. I've often thought it would be cool if there was some way for e.g. your version control software to be tied to the code you write that passes certain test(s), such that you could somehow verify that without that code your test fails and with that code your test passes. For the present though, TDD tends to work pretty well for me.
Labouring under the idea that Tests Must Come First (and everything I've seen, and everything I /do/ see now suggests that that is the central idea in TDD - you write a test, then you write the code to pass it) without pivoting to see that testing is a useful practice in so much as it helps developers is the wrong approach.
The problem is that ideologies like TDD try to come up with simple rules for how to develop (i.e. always write tests first). In reality, it's better to use your brain when you're deciding how to approach something (i.e. is this a good situation for writing tests first?)
I often code three or four commits worth at once when on a roll and it's a pain doing multiple staged adds, stashes, tests, and commits to store each sub-change on its own. Having a tool to help with this would be great.
When you write something you run it to make sure it works. Bottle that and call it a test. And then merely do this before you write the code. "What will I be done when I get this code to do?"
Don't plan, and spec, your whole API before you've written a line of code or anything, but do write a one-liner to check whatever it is you're going to do so you know when you're done.
Ideally you'll write a test (still just a one-liner) for every corner-case that you cover in code, because you need a failing test to know if your fix actually worked.
And face it, if you're doing "real" development work you've got business goals, and a spec with features intended to deliver on that, and you need to cover every tiny detail. If you don't have the spec ('fail on values over n') directly linked to the code that implements it, it will get lost, and if you aren't testing it you can't legitimately claim to be delivering it.
A spec, such as you'd get by performing useful TDD, is a requirement for modern software development. You could write it all up-front in the planning phase (waterfall style), or all after in the testing phase (waterfall style again), or you could interleave it with research and development, which if you think about it, sounds kind of agile by comparison...
For example lets take a simple newsletter signup form with only a field to enter an email address.
Now you shouldn't write a test to see what happens when you enter an email, a wrong email, if the confirmation email is sent, etc. Start with the simplest case: "If I fill in my email, and I press klick, then it should show me a confirmation that I singed up." Not more. Now make that pass, and you have working code!
Now you can add tests that verify that a wrong or empty email is rejected, now make that pass. More working code!
With TDD you should never end up with a lot of tests and no code that solves the problem. In that case you are actually writing too many tests and not enough code.
This is useful because you're constantly making and checking hypotheses: "This code should make a test fail in X way... yup. Okay, now this code should make the test pass. Yup. Now this change shouldn't change behavior, and the tests should still pass. Oops! Undo. Oh, that method has to be called in the constructor. All right, this change should work. Good."
(If you're interested, my book goes into more detail, and it's online: http://jamesshore.com/Agile-Book/test_driven_development.htm... . I also have a extensive screencast of real-world TDD here: http://jamesshore.com/Blog/Lets-Play/ )
Despite the extremists, TDD really is a useful tool--and that's all it is. Sure, TDD has been turned into yet another vehicle for programmers to engage in religious wars, but you can't really blame TDD for that, any more than you can blame Richard Stallman for emacs vs. vi wars.
Don't just write all the test for every little function in the beginning thats stupid, How do you even know what helper function you need to implment.
Achieving this, in my experience, comes down to disciplined used of mocks and stubs. I've seen people lean too heavily on them, or not heavily enough. The biggest problem I've seen, particularly common in the Java world due to jMock's datedness, is over specifying expectations on mocks. Stubs which can provide canned responses (whether through a framework or manually rolled) are absolutely and totally the most underused yet useful tool in your testing arsenal.
Perhaps I'm doing it wrong, but I've found in my experience that bugs I miss in testing increase proportionally with my use of mocks or stubs. Invariably, when I'm forced to stub something and replicate its behavior I miss something subtle that comes back to bite me.
First, people rely too heavily on mocks/stubs/fakes/(whatever you want to call them). This has gotten better in the past 4 or so years (in my mind, largely because of Rails and it being "acceptable" to hit a DB in a test - though it might actually predate Rails). I think this problem is pretty straightforward to understand (again, especially when you look at the Rails way to test a DB interaction versus a more "traditional" way).
However, there are instances where mocks/fakes/stubs are important. Some outside dependencies might not be accessible during testing, might not be predictable/deterministic, might be too slow or might simply involve too much setup. Also, solely relying on "real" implementations can also make your tests too brittle. Why should the logic for can_legally_drink? break if the DB column is changed from DOB to dateofbirth (ok, that's an extreme and poor example, but you get the idea, hopefully).
Anyways, assuming you agree that sometimes a fake is simply better, you get into mocks vs stubs. To me a stub is dumb and a mock is strict. Stubs also automatically reply with canned answers, mocks don't. A mock is used to assert that a certain expected call was made. A stub is used just to get your test to move over a line of code.
The problem is that people use mocks over and over again. Re-specifying the same expectations in every test..making X tests break when you change the behavior of the interaction, versus just 1 test. As semi-pseudocode (and yes, it might be better to just hit the DB in this case):
To me, you kinda wanna check that the above code properly interacts with someStore. This is when you use a mock: You also want to make sure the password matching works. This is where people go wrong. They'll use a mock (strict) again..which'll just repeat the above code...except this test has really nothing to do with how store...we just want to get over the line of code: If you are familiar with jMock, it's kinda the difference between allowing and oneOf. oneOf is very strict, allowing isn't. You really should use allowing whenever you aren't explicitly testing the interaction (and you shouldn't explicitly test the interaction in more than 1 test (dedicated to testing said interaction)).Some framework even return smart canned answers. So instead of returning nil, they'll default to a new instance of the type, or an empty array, or some default scalar value (like an empty string). This is particularly useful when you just want to get passed a null reference (which is exceedingly common).
Does that make sense?
I can't seem to find a blog post discussing the merits, but IME strict mocking leads to a lot of tests breaking just because you added one more piece of functionality to a method. Now, this causes you to edit all sorts of existing tests to cause them to ignore a call to collaborator. While you can extract common expectations and such to eliminate duplication, you still end up with many tests failing due to one small change in behavior.
Normally your tests tend to assert one or two behaviors per test. With strict mocking, your tests become awkwardly cluttered by setting up expectations/interactions for things that are unrelated to what they are asserting.
With nice mocking, it's much easier to get away with only the expectations/interactions that are directly relevant to what a single test is asserting. This also means that you get more fine-grained failures, which speeds up diagnostics.
That's an interesting question - would a unit test prevent this error? To answer this, let's consider what happened: the database didn't respond in an appropriate way, and probably returned an error. I'm not exactly sure what exact error was displayed, but WordPress has caught this situation, and handled it the best it could - displaying a proper error. I would write a unit test for this exact scenario - using a fake (mock) object, simulate the database timeout, and make sure that the error was caught and handled appropriately. Other then that there's no real other way to gracefully recover.
When writing code it is imperative to to get great feedback, as often as possible. In a number of cases, TDD helps you do that. In cases where it don't, well, it doesn't, so don't use it.
Don't get me wrong, I'm sure that all these guys and chicks that live and die by TDD are smart (probably smarter than me), but what they are building is starting to get more and more out of touch with real-world requirements (I wanted to say somth about how this resembles a schizophrenic world, but for the moment I'll stop short of that).
My father was a civil engineer, at a time when they didn't have AutoCAD available and all that fancy computer stuff. As he elegantly put it: "if I mess things up in my work people will die, either in 2 weeks or in 20 years, when the next big earthquake will hit us". Well, I never saw him "testing" building apartment-blocks, or industrial buildings, or roads, he made sure that what was on the sheet of papers where the construction plans were drawn would get built as accurate as possible in real life. And keep in mind that he was managing construction workers, many of them close to illiterate, even former convicts, and not CS-graduates. That's the job of all civil engineers from all over the world. So, if they can do it, why in the name of God do we programmers write tests for mundane stuff like sudoku-solvers? Something is not right.
At the point where you're writing tests to impress other programmers (see article) about how much of a 'craftsman' you are, you lose. When you're writing tests that help you as a developer, and help get the software out more quickly and with higher quality, then you're doing it right.
Testing doesn't help you come up with good algorithms. It can help you with not messing up an implementation, though.
It is not a coincidence that one of the reasons TDD'ers love rspec so much is that the "code reads like documentation" - it's even called r spec.
Perhaps that is ok for mature projects. But I often hear TDD and agile used together as if they weren't in conflict and I get confused. Then again, I've only ever worked on new projects.
Maybe I'm missing something?