Test Driven Development Actually Works (blog.jphpsf.com)

You often hear people advocate that writing tests (unit, functional, etc.) when building software leads to less bugs in production. In this post, I am trying to demonstrate this statement by analyzing real numbers from my current job. No tests were written originally and once tests were included, I became a strong advocate of the TDD methodology.

My team has been working toward this goal for the last 8 months. We've talked a lot about testing and making it a requirement to have passing tests before a feature can be called "dev complete".

In going back and writing tests for even some recently developed features we have found a few (albeit, minor) bugs. In adding tests to legacy code we found mountains of issues, but it's also very time consuming to do.

We're just starting work on an API and our goal is to have tests written before the code is written, allowing us to essentially write the requirements for each API and write the code to match those requirements. I'm looking forward to the process as I really feel it'll make our code clean, concise, compartmentalized, and much easier to maintain in the future.

It would be nice if it would be true, the graph looks great and props for supplying the data. Correlation does not equal causation though. Are there any scientifics researches on TDD or any programming styles?

I am a latecomer to TDD and try to increasingly do it in all of my personal projects, no matter how trivial.

I can't say how much it has been objectively beneficial because I haven't taken the time to measure metrics and I willingly submit that I may be just a sloppy programmer overall. But metrics aside, I will say that TDD, when I've done it on a "sure, why not, there's no deadline", has had great benefit to my morale and productivity because of how it builds the habit of programming into me.

Not the habit of TDD, but the habit of programming itself. Just as the cues, triggers, and rewards of a slot machine hooks people into gambling even if those people dislike gambling, TDD helps me break out of the "I really don't feel like programming right now" mentality that I've always had. When all I have to do is solve some small tasks -- with the knowledge that a reward is immediate (the passing of the test), it's easy for me to jump into it...and once in awhile, I'll even have the "just one more test to pass" attitude.

Now obviously, this (like slot machine addiction) is no good if you haven't built a good project plan and/or lack some sense of ingenuity. But even then, writing some a series of menial functions is much preferable over doing nothing...just as running a pathetically slow two-mile run just to meet a running goal even though you feel lazy is way better than descending into self pity/shame after you drop another new Year's resolution.

I could also point out that TDD has really helped me practice orthogonality but I think it's enough to say for now how it has improved my attitude. yMMV

This experiment is not very rigorous. The Q4 2011 change in the proportion of UI bugs could be caused by the addition of back-end code with little UI (such as maintenance tasks), or by an upgrade to the bug tracking software that encouraged grouping multiple UI bugs into one bug report. And the author didn’t calculate whether TDD slowed down their work – it is not clear that TDD is better if it avoids defects but also increases development time.

You can find more rigorous studies on whether TDD actually works on Google Scholar: http://scholar.google.com/scholar?q=test-driven+development+...

I'm afraid the OP is jumping to conclusions.

You can't arrive to the conclusion that TDD reduced the number of bugs based on a percentage ratio between UI/server bugs. You could simply have a bigger share of backend bugs over time.

You need to see it in absolute numbers, using some meaningful metric (bugs vs. LOC maybe?).

Is TDD supposed to prevent bugs? I thought the main point of TDD was it was suppose to encourage better design.

First write the test. This is supposed to make you think more about the API and how you'd like to use it. Then write the implementation now that you've effectively tried using the API by writing the test. That why it's called Test DRIVEN Development. If all you're doing is writing unit tests that's not really TDD, that's just writing tests.

TDD works for some things, not so much for others. If you know what you want to build, and how it will work, TDD makes it more likely that you will build it correctly, and that the programmatic interfaces will be pleasant to work with. If you don't know how to make the thing you want to build work, TDD is not going to help you. At all.

As an illustration see http://devgrind.com/2007/04/25/how-to-not-solve-a-sudoku/ for an amusing example of TDD failing horribly for a simple algorithm problem.

Perhaps I'm missing something, but it seems like what this post describes isn't so much "test driven development" as "testing". Rigor of the data analysis aside, unless I misread, it seems like a comparison between not testing at all, and testing (I didn't notice any specific mention of TDD methodologies).

Shouldn't the comparison be between full-on TDD (e.g. write tests first, etc etc) and 'old fashioned' testing (for lack of a better term)?

The author seems to emphasize about practicing TDD and that it will produce LESS bugs which is not what TDD is all about. TDD is about designing your architecture doing your "tests first".

On the other hand, some developers can write code first then unit tests later which can also produce less bugs but this is not TDD.

Personally I don't find the idea of 'test first' to be very practical in the kinds of tests that I write. I think the pragmatic programers used the expression 'spike and stabilize' to describe a pattern of development whereby devs get something working then write some tests to support that new feature. They made an analogy of mountain climbers free climbing for a bit then securing everything with the ropes. In order to do 'test first' I would have to decide things like what to call an input field on a web page so that I could write my failing test then code the page. That is too early to make decisions like that. Also, when people say they write a failing test then the impl. are they really writing a single failing test. Often I find I need several tests to cover all the cases. I bring this up because I'm a big fan of TDD but it seems too easy to poke holes in the whole 'write the failing test first' approach.

A general note on TDD: I love it.

TDD encourages better APIs and more testable code. I don't use TDD to prevent more bugs. I don't even know if this is really true.

Everytime I think of a new feature I fastly have an API in mind. Starting with a test almost always shows me a better way to provide the functionality (easier, more failsafe). Thats one of my two key features of TDD.

The second key feature is that the tests you write are much more expressive than tests you write afterwards. I observe that everyday in my team and company. Writing tests after the implementation is done leads to a lot of mocks and just a static verification that the implementation works how it works. In the end, it is difficult to change the implementation and easy to break the specification. Writing your test ahead of the implementation means you test the specification and not how you implemented it. Tests become much easier to read as they express the specification instead of a 1000 lines test where every variable assignment of the implementation is checked.

As a last hint: I read here that people complain that TDD does not work with "prototyping". I agree with that - the problem with that statement is that a lot of people use their prototype as productive code later on instead of rebuilding it after the experimental phase. In my opinion that is a misuse of the word "prototype". A prototype is an experimental work to verify or learn some ideas and afterwards designing a better system. Other people may state that testing everything is not possible - that there are situations where it is not possible to write a test. I would bet that in 90% of that cases, this is just wrong and means a lack of testing experience of the writer. I can really recommend the following book: http://www.amazon.com/Driven-Development-Embedded-Pragmatic-... I rarly do embedded system programming. My main focus are languages like Java and Go but this book is worth its money independent of your choice of programming language.

The article is not conclusive on whether it was TDD or just plain automated testing that provided the benefits.

I've heard good arguments against TDD, but I haven't heard any good arguments against automated testing. (Other than situations where automating a process is cumbersome, like hardware bugs.)

There are 2 major flaws in this experiment - both stem from this assumption

that the first couple of years of working for this company, are equivalent to the next 3 years.

This is obviously NOT correct.

1. The more you work with a codebase, the more you undestand its details and complexities - this alone should help you eliminate more bugs during developement.

2. You will have improved as a developer - meaning that the last 3 yeras in your life will have more experience and knowledge than the first 2.

As sad as I am at stating this fact... you cannot conclude that TDD actually works, because there are too many other variables involved.