If you write testable software and actually write the tests, the end result is the same whether you test first or test later. You're designing with testing in mind and creating useful tests either way.
It's really a matter of code that lacks tests that's an issue and especially of code which isn't designed with testing in mind. I think they overinterpreted the value of TDD as simply test-first. Test first can be good but it's more a motivational tool and a productivity simulator (yay look at those green checkmarks!) than a real benefit.
The end result is identical, this sort of craps on it as a motivational tool too if you consider that only as a factor in development time at least. It'd be interesting to see developer satisfaction included in some way too.
I have always worked in a "test last" mode. That's the way I learned to program. Write some code, test it. When it works, write some more code, test some more.
My attempts to work in a TDD style ran up against decades of intgrained habits and I never really found it satisfying or natural.
It took me a year or so to make the transition. And I think the transition is easier to make in a greenfield or already-TDDed code base.
That's not to say that you should try it again; I like TDD a lot myself, but software methods are so interdependent that I think each person has to judge what works best for them.
> If you write testable software and actually write the tests, the end result is the same whether you test first or test later.
This is definitely not my experience. As Kent Beck says, TDD is a design technique. It forces you to always start thinking of the code from the outside of the unit. If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implementation. With test last, I'm also less likely to test everything well; after the implementation is done, I believe it works.
If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implementation. With test last, I'm also less likely to test everything well; after the implementation is done, I believe it works.
Will you?
Are you sure?
Do you have data to back your supposition?
My contention would be that at the end of the day, the requirements of the interface to support unit testing will result in a very similar set of design choices, whether you write those tests up-front, as-you-go, or after-the-fact.
The only difference is that if you write them after-the-fact, you may push some amount of refactoring to the end of the process instead of doing it along the way.
But I'll bet you have about as much data as I do to back your beliefs. ;)
That's my experience. I started doing TDD more than 10 years ago, and it took me about a year to fully make the switch from test-after to test-first programming. I regularly try experiments with different personal code bases.
If you're arguing from personal experience, that your code ends up just the same either way, good for you. But I suspect you're arguing from theory here.
Perhaps it depends on how you implement each. If your idea of test last is writing a full large module and then trying to test that all at once, you might wind up with a different result than if you wrote an individual function or two and tried to test only those at once.
Getting the testing as close in time to the implementation as possible probably makes a bigger difference than first vs last.
I definitely agree with that. If there is somebody who actually writes a few units of production code, a few units of test code, and then refactors, then I expect things would be pretty close.
However, for me that's a hard set of behaviors to stick to. If I let myself say, "Oh, I'll test that later" I'm likely to get lost in the problem, write bunch of production code, and then say, "Well, it already works, but I guess I'll add some tests." There's no clear stopping point.
Test-first programming, on the other hand, is easier for me to stick to. If I don't have a broken test, I write one. If I have made the test pass, it's time to write another test before I write more production code.
Why would we assume TDD would improve code quality or development time?
If, at the end of the day, the result is the same volume of tests and the same level of overall coverage, the order of activity would seem to make no difference. What would lead you to believe otherwise?
The real flaw in both methodologies is that it's the developer checking their own work.
I'd make the claim that if you really wanted to do it "right", you'd have a specification, one developer writing tests to the specification, and a different developer writing the code to make the tests pass.
Of course, that's probably not workable in practice... make for an interesting experiment, though (and yes, I'm too lazy to search Google, only to discover someone else already came up with the idea).
Actual studies were never needed to convince managers to switch processes. Bonus points for blaming old problems on old process while blaming new problems on "not doing agile right".
Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc.
But I know some people are wired differently; they'll think a lot more about scenarios first, then code after they have everything accounted for. For them, TDD as a philosophy seems more fitting.
I think the chasm exists between _untested_ code and code that has tests. I've never understood the seemingly-religious zealotry behind TDD as an XP practice. Just like pair programming... if it works for you and your coding style, awesome. But don't force it down my throat or act like it's the One True Path to clean code.
Well, things start to get complicated and messy when the team size starts to grow and the technical debt starts to mount. Thats when it needs to goes from hacking to engineering.
Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth between the two testing methods even more rapidly.
When doing UI code (with something like React) it's next to impossible to do TDD - you can't know the structure until you actually implement it.
If I'm fixing a bug in some sort of non-ui code, I often find TDD to be very helpful. Create a test case that demonstrates that failure, then fix the implementation to make the test pass.
That's an interesting discussion and article. I sort of do this already for new features. You really feel the power of it, when after writing a lot of stuff you realise that there's a hitch, rewrite from earlier and shudder at the thought of the mess and wasted time if code had been written.
In its final state the document becomes a series of steps that I can check off, which is helpful in a motivational sense.
I had thought of involving users in the design of new features via the support forum but I'm afraid of how that might turn out. A readme type document would probably be about as good a way of doing it as any though.
I find the same thing. It's almost like writing an essay, I need to sketch out an outline first, which is often getting just far enough to convince myself that the approach is reasonable and there aren't any surprises ahead. Once it's time to crank out the details of individual sections, the work often starts to feel monotonous and that's where TDD can be invaluable for helping me to stay focused and comprehensively work through all the edge cases.
> they'll think about scenarios first, then code after they have everything accounted for.
I'm one of those people. But I find it hard to believe that a test suite can reflect everything I understand about a problem domain. Focusing too much on unit tests makes it difficult to find the general rules that govern the behavior of a system. Have too few unit tests, and they won't capture all the subtleties and corner cases. Have too many unit tests, and they won't fit in your head.
> I think the chasm exists between _untested_ code and code that has tests.
IMHO untested code isn't always a bad thing. Anything related to privacy, security, or data integrity should be heavily tested.
But beyond that, code should need to earn its tests. (At least in a web startup context.) After all, the core of agility is being able and willing to go in a different direction when something isn't working.
I think there are a lot of nuance to this statement. After a few years on the start up scene, I believe that if you are using a "duck" typed language your core concepts should be tested without exception (for example rails and django models). The farther you get away from these core concepts, the less your tests need to be there and the more likely that code is to change.
In addition, if you are truly in the earliest stages of your startup (pre-revenue/traction) the tests are not needed -- but you have to be _very_ aware that you are making a tradeoff for sheer velocity and you will have to pay the price later on. I've seen too many startups fail to pay the price and it comes back to haunt them and reduces their overall velocity.
If your code doesn't have tests then you can't refactor because you don't know if you broke something. Getting the right level of tests is important, they should check the interface and not the implementation, otherwise I agree, you can't refactor anything. But if you test your interfaces the same way that real code would use it then it actually makes you more agile, because it gives you the peace of mind to do major changes.
"Tests can make refactoring much easier" is a statement I could easily buy into.
"If your code doesn't have tests then you can't refactor" is pure dogma and trivial to falsify (go on, refactor some untested code now!). More likely to introduce bugs? Perhaps. But relying on that absolutist statement about refactoring will alienate me every time.
80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions.
So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).
Naturally, I never get this part right at first, and so I wind up refactoring a lot as I go along. ("Build one to throw away", I believe this has been called.)
Now in my experience, TDD requires me to refactor more during this process than tests-after-code. But suit yourself. Changing the order within the last 20% won't be earth-shattering one way or the other.
This is why designing the interface in a language with a powerful type system provides the same quoted benefit of TDD. It helps you think about the interface before you move to implementation.
In my experience, using a type system to do this requires much less effort and refactoring.
Unit tests can demonstrate that my code is correct; it can say nothing about how my code is used by others.
In contrast, a type system can extend those protections downstream.
(Yes, yes, unless you go way off the deep end of dependent types, you still need tests. But a powerful type system can systemically prevent a huge number of very common bugs.)
My take it on it is that it can remove most of the boring bugs you would get in a less strongly-typed language and leaves you open to the interesting bugs (eg, logic bugs).
As for dependent types, I'm not sure that even then you would be able to ensure that, say, your cache is correctly invalidated.
I've found implementing test as I go along writing a new API help code quality and speed in two ways: I catch my own logic bugs early and the self-feedback I get from writing the tests helps me to write a more usable interface.
I never catch all the design errors with non-coding design work - this gives me the high level view which is critical - but there are always some logical errors and bugs I've failed to model correctly which early tests catch.
But yeah, the optimal design-implememt-test probably depends probably very much on the implememter and on the problem to be solved.
The third meta-advantage is that if I'm writing hobby code for myself and the profject is large compared to the weekly time allocations available in my sparetime I get something to help me remember where I was (say a month back) and gives me something to compile and run immediately (which is a huge motivational booster for me).
> 80% of software development is designing correct abstractions/interfaces/APIs
Not sure if that claim is wrong or not, but I will state it anyway. If you write in TDD manner and you decide to change the API/abstraction a little bit because of something you didn't think of, you will have to correct both tests and the code. When you do test-after-code it's much easier (IMO) to hit correct abstractions and interfaces because you are working on a living thing, not some artificial test cases you have to figure out. Then you write tests to ensure those abstractions are respected and in working order.
TDD people, please correct me if I'm wrong.
EDIT: disclaimer: I'm a front-end developer working mostly in React and Redux
> But don't force it down my throat or act like it's the One True Path to clean code.
When I was a programming beginner and started learning about TDD, I was always thinking how soon to start using it because everyone says it's such a good idea. If you would search anything about it, you'd rarely find a resource not marvelling at its surprising benefits (at least, that is how it was few years back). Granted, it may have its merits but is it really a good idea for everything? No way!
One of the anti-patterns I have seen in programming world is how everyone jumps on adopting / advocating a 'good' practice without understanding how it fits the project. I wrote about this a while back[1]. Cue in — git flow, TDD, not using goto, DOM Parser instead of RegEx.
To me, it depends on the problem. With some problems, I'm not sure yet how to implement it, but I do already know what the result should look like. That's an excellent case for TDD. You get to work on the problem, refine your ideas about it, and you start with the part you already know: the results.
If you have no idea what the results should look like, then TDD is pointless.
The "lightbulb" moment for me was doing what Kent Beck suggests, and writing down the list of tests I "knew" I needed to write, before writing any tests. That's a very similar step to diving into writing the code first, "knowing" what it ought to look like, and the ideas feeding into either step would be the same.
Writing a list of tests first is quicker, though, and working through the list, updating it as I learn more, tells me that my initial ideas were wrong often enough that I'm almost convinced that the true secret of TDD isn't the tests at all. It's that checklist, and the purely mechanical process of working through it, checking them off one-by-one and fixing it up as I go along.
To relate this to your situation, I often find that I'll only get started with a fairly fuzzy idea, and the checklist will be short. I can still expand it as I go, knowing that I'm working on as solid foundations as I can given the constraints.
I think there's a danger in writing off practices like TDD with "if it works for you..." because it avoids discussion of why it works for some people and not others. That in turn gives people license to ignore it simply because it's unfamiliar, and the brain doesn't like unfamiliar things by default. That's a shame, because for some of those people they'll be stuck in local optima, writing off the one thing which could break them out of it.
>Could it be that TDD vs. tests-after-code is a highly personal thing? I personally find it easier to write good tests after I've coded something functional. Before hand, I know one or two fuzzy ideas of what I want to accomplish, but I can't list out the concrete, real-world test scenarios until after I've coded something, poked and prodded it, etc.
I generally find it better to engage in "top-down" development - specification->tests->high level code->low level code as it's cheaper to fix mistakes at higher levels first.
If I'm doing purely experimental code though - where I'm not sure of what I even want or what the output should be, tests are just a waste of time.
It also loses its effectiveness the more declarative the code is. If I'm essentially tweaking configuration or HTML there's no point.
When I first learned about TDD I thought the same (order of test and implementation does not matter that much). Yet when you do the TDD steps
* write a failing test (RED)
* implement minimal functionality so all tests pass (GREEN)
* REFACTOR
and overcome the initial revulsion you will get a ton of benefits that you didn't know existed. I'll first give a few that are more on the code quality side
* You have a complete test suite, without redundant tests
* Tests are usually not that mock-focused [] as the implementation-after-test didn't introduce random roadblockers for testing.
tests do assert the relevant properties (tests-later code bases often have code bases that don't assert much or properties you don't care about).
* When you write your test first, you basically write a small example how it feels like to use the API you have in mind for solving your problem. This drastically improves quality.
Some non-technical benefits of TDD
* I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing.
* TDD is programming gamification. You are rewarded with regular, small successes of having added another test and made it green. So whenever you ask yourself: OMG, have I made any progress in the last 3 days, you can actually tell: Yes, I added 15 tests and made them pass - I objectively got closer to implementing that feature.
* Not an issue where I work, but have often heard about this from other places: If tests are there first, no one can tell you to not spend so much time on writing tests. TDD-style is protecting your boss from shipping that car without having the brakes checked.
* The same line: TDD gives explicit room for refactoring.
[*] obviously one cannot avoid mocking especially when dealing with resources external to your code base, yet I have hardly seen TDD-originated tests that mocked
> I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing.
This made me think. There are many axes of software development, and aside from TDD/non-TDD axis, there's also the bottom-up vs. top-down axis. By top-down, I mean you start with a skeleton that provides an interface to the outside world, and the fill in the implementation details. Bottom-up, on the other hand, means you first write some routine that you know will be vital in some form or another, but without immediately seeing how it will be hooked into the interface to the outside world.
Top-down has the advantage that you can have something runnable quickly, and it seems to me that TDD requires top-down development.
But somehow, there have been times when I was working on a piece of code and it just flowed more naturally in a bottom-up manner. It has happened that I went for a day (and very rarely longer) working on tricky algorithmic code before I even tried to compile it for the first time.
This is definitely something that I try to avoid when I can, it's just that sometimes it works out that way. And it is totally incompatible with the TDD way of doing things.
How often this happens surely depends on the type of project you're working on. As usual with these things, it seems like TDD can be a useful inspiration, but it shouldn't be taken as dogma.
I dont think writing Tests first makes you work top down or bottom up.
If you want to work bottom-up (I like to work bottom up) you just start by writing tests for your bottom up functionality.
And of course I don't want to impose TDD on others, but I would like others to try it out sincerely (recommending kent Becks book for this. Don't do it without guidance, many people who reject TDD after trying it out were doing some kind of Test-first method, but didn't get the benefit because they were not aware of stuff like get one test at a time to complete, etc.)
---------
> shouldn't be taken as dogma.
Life in software engineering becomes much more relaxed when you realize that every piece of advice and technology is in fact based on dogmas and is not based on sound empirical, statistically relevant evidence :-) (apart from the pure math part).
It is just really really hard to study software quality in an objective way that satisfies scientific criteria. So we are kind of stuck with hearsay in IT
So TDD is kind of dogmatic. So is OOP, or the rejection of gotos.
I think test after is a reasonable approach. I am constantly dealing with new code bases and in the interview process I am often asked about my philosophy about testing. My response is thatbtrsting occurs on two fronts from the top down ( functional UI tests) and from the bottom up (unit tests). When it comes to unit testing my approach is to focus on the hotspots. If something gives you trouble, or if you find a bug/ issue then wrap it in a unit test. That way you don't have to worry about it. Bugs tell you where the weak spots in your code base are. When they speak to you, listen and take some action. Otherwise, I feel like chasing blanket coverage is not worth the effort in most products.
I think test after is a reasonable approach. I am constantly dealing with new code bases and in the interview process I am often asked about my philosophy about testing. My response is thatbtrsting occurs on two fronts from the top down ( functional UI tests) and from the bottom up (unit tests). When it comes to unit testing my approach is to focus on the hotspots. If something gives you trouble, or if you find a bug/ issue then wrap it in a unit test. That way you don't have to worry about it. Bugs tell you where the weak spots in your code base are. When they speak to you, listen and take some action. Otherwise, I feel like chasing blanket coverage is not worth the effort in most procure.
>Seems to be a common misunderstanding around here that you, too, have fallen victim to...
I work with people who openly believe that unit testing is a waste of time. Personally, I think they are afraid, because they don't even know how, and don't care to learn. They would see that title, and that's all the confirmation bias they would need.
Err, my apologies if that came across as a slight, I literally meant you and a lot of other folks made the exact same mistake. No judgement intended, it's clearly a common misapprehension.
My guess is that a lot of folks got introduced to unit testing through test-driven development and, as a result, conflate the two, assuming the former necessarily implies the latter.
Speaking for myself, I was writing UTs long before the TDD fad landed and so I never picked up the habit. It's just not the way my developer mind works, and I've not found it compelling enough to try and re-train myself.
That said, not believing in the value of automated regression test (of which UTs are the lowest hanging fruit) is utter madness...
My anecdata matches the author's - I feel more productive doing TDD.
Perhaps because it's less stressful. You think about system design as you code, instead of only when you hit a wall and have to rewrite everything, or when you have to clean up for code review.
Either way, if it has little to no compact on dev team or code quality, I bet the positive impact TDD has on team morale would make it worthwhile.
TDD has a very specific meaning. It means you write tests, then write code that passes those tests. That specific order. If you're not doing that, you're not actually adhering to the definition of TDD.
TLD could mean, for example, writing a module, class, or function with a defined interface that implements the contracts for that interface, then writing the suite of tests to validate that module, after which you move onto the next module.
Strangely enough, you can do that while you're developing the product without adhering to the process dictated by TDD.
My experience has been that TDD is worthwhile when working with notoriously slippery whack-a-mole functions like handling time or money. The time saved by catching regressions vastly outweighs the time taken to implement the tests.
In contrast, TDD has been a waste of time for me for UI-based work, as the effort needed to properly expose the functionality under test is too great and the requirements and design change too quickly to be worth it.
In the latter case, writing some deterministic UI tests against mock data after the requirements and implementation have settled has proven much more effective in preventing regressions.
That means you can't use coverage tools, so this approach makes the tests unquantifiable. It is not how most major dev shops work, because of the bean counters and the "senior" developers/leads/managers/... who seem to want nothing but appeasing beancounters.
It also produces superior software in my experience.
Automatic test generation tools. There's quite a few on github, but the real ones are the ones you write yourself. What you're looking to do is to run the software a few times and record tests for particular pieces of code during those manual tests.
Can you elaborate more? Im a frontend guy and sometimes I write JS plugins for the browser (that run after page load, and apply based on how things render on the page, or bawed on user interaction with the page) and non-frontend folks tell me I need tests for my code, or dont want to look at it until I have test. How would I build a tewt for this, other than a functional test that runs in-browser and the test is whether it works or not?
Here are two strategies. First, you can go "outside" what you've built, simulating a user's interaction and examining the resulting DOM. In general this is a tool-centric approach.
Another approach is to partition your code into what Gary Bernhardt calls an "imperative shell" of code that touches the outside world and a "functional core" that does not. Then unit test the functional core, which shouldn't require special testing libraries, and validate that the imperative shell works by the occasional manual or "outside" test.
First off, if your code don't have bugs, then that is preferable! In my experience with front web dev the issue has always been different (old) devices. Testing could be done by browser macros that simulate input, like mouse movement, click on x, y, z, etc. Then take screen-shots to see if something is off.
You can also make your widgets throw errors (1) and call home via XMLHttpRequest if an error is detected.
Exactly my experience - I've done TDD for 3 different types of calculation functions in 3 different languages over the years, and all were beneficial: financial risk calculations in C++, report data calculations in PHP and billing calculations in JavaScript.
On the other hand, TDD for operations that involve I/O (including user interactions) were not helpful at all.
I never really viewed TDD as better at reducing bugs for a short term project, its going to have marginal better chances of getting additional test cases.
I view it more as important for breaking the growth of testing effort in an iterative project. With each release the scope of what should be tested to fully test a project climbs and unless a team wishes to linearly increase the size of its test team its all but certain tests will be skipped.
TDD gives us the ability to always full regression test as its just machine time. Its a safety factor in knowing nothing is broken which in turn gives us confidence we can refactor.
Therein lies the rub with all the faith-based modern development practices. Getting results? That's the power of The Practice. Not getting results or seeing negative impact? You're just not doing The Practice right and/or hard enough.
This is the kind of stuff that in the aggregate you can't show a relationship, but I bet if you controlled for type of project one would see some interesting results. Anecdotally, I know some firmware engineers that shit out the buggiest code I have ever seen, and test driven development would have definitely improved the customer experience. Because when the engineers have literally no tests other than trying stuff out with a printf on the target embedded device, any amount unit-testing will wind up helping.
I just came out of an embedded system-sy project, and I did have some tests for my ringbuffs, and sprintf.
But tests for any of the interactions between subsystems is quite problematic. And then testing on the device might be problematic due to space constraints, but testing on a simulation is also problematic...
I don't know how you'd realistically test it.
Tests before/at/near development time really helps your code design - I've seen how ensure code is unit testable simplifies and enforces layering, etc. Really disagree that this does not help code quality.
There is a problem with all these studies - they all use a very small amount of programmers (21 in this case) with no experience (all graduate students in this case) and presumably no significant experience with TDD or TLD.
I'm not making a stand about TDD here - I just think we need to have much better computer engineering science studies if we want to have significant results.
Agreed. I also would question the short time scale of the test. It took me a year or so to really get good at test-driven development.
I also think of TDD as a sustainability practice. If I'm writing a small thing that I do not intend to maintain, I won't bother with TDD (or with tests at all). But I'll definitely TDD something where I expect to come back to it frequently, especially when I initially don't know the requirements, and I expect requirements to change over time.
In practice, I suspect a lot of the interesting questions about software are effectively unanswerable with the budgets available to CS profs. I can't imagine really answering this question without doing something of the scope of a substantial medical study.
Really don't know why you're being downvoted. Grad students at diploma mills are awful programmers, and even those in real labs are good at putting out proof of concepts, not production code.
The problem that I have with this article is how people will interpret the results. The test is comparing (presumably) Comp. Sci. graduate students who already know good design patterns, best practices, etc at a relatively high level to see if they are faster and more accurate by testing before vs. after writing the main code. (TDD vs. TLD)
That's all well and fine, and possibly completely accurate. However, many people's takeaway is going to be the out-of-context & incorrect title of this post. (It does not say TDD is worthless - it says it is essentially the same as TLD)
I've always looked at TDD as a tool to help push less experienced, less "educated" developers into 1) even using tests at any point of the development cycle, 2) creating tighter, cleaner and MORE TESTABLE code by the time they've reached the end of the cycle.
So, if your team is and always will be well-educated, experienced programmers who already understand how to always do everything correctly from the beginning, feel free to use either method.
Really? Almost all of the post-graduate CS students I've known are experienced people who went back to school because they were at the point where they needed to either move into management or become a very high-level expert.
That's only personal experience, however, and mostly out-of-date
That said, I would hope any of them would be a step up on the new programmers from the "churn 'em out" short courses many are forced to take as the only affordable way they have to get into the industry, but then are left with little knowledge beyond how to get the requirements fulfilled.
Yes. I can say 100% that the major difference I see in hiring interviews between grads and laterals is coding ability. Even at PhD+1 it's vastly improved
This is a misleading title and conclusion. The study showed a huge benefit of TDD over Waterfall, and it is only when compared to ITL that it was found to not be better.
But moreover, I think it's important to understand why Beck pushed for TDD.
TDD is like saying "I'm going to floss before I brush every time, no matter what."
But, when people don't do TDD they typically aren't all saying "I'm going to brush and floss afterwards every time, no matter what."
Instead, most say "I'll floss regularly at some point, but I don't have time now, and it takes too much effort. I'll floss here and there periodically, maybe before my monthly meeting or big date night."
Another reason Beck pushed for TDD was method and solution complexity reduction which results in lower time and cost required for maintenance because code is simpler to read and understand. Again, with ITL, you're still writing tests for everything, so you'll see those benefits. However, if you fail to write some or most tests, some developers will write overengineered solutions to things and have overly long difficult to follow methods that will make maintenance suck more resources.
If you want to go beyond this study, though, Beck, Fowler, and DHH had a critical discussion about TDD in 2014 that's worth checking out:
Not only that... when you test the efficacy of medical interventions the gold standard to strive for[1] is not whether the new intervention is better than placebo, it's whether it's better than $CURRENT_BEST_KNOWN_INTERVENTION. I suggest we should be aiming for a similar standard in testing software engineering methodology.
I think it would be very hard to argue that Waterfall ~= $CURRENT_BEST_KNOWN_METHODOLOGY.
[1] Of course, this isn't usually what happens in practice when pharmaceutical companies are doing their own testing, but it's what should happen if you actually care about efficacy and not just PR/sales.
The extra complicated architectural refactors I've seen done in the name of 'test-ability' have been eyebrow raising. TDD isn't a guarantee in inducing engineers to KISS. You can still make overengineered crap tests first / TDD or not
TDD is king when refactoring, or proving an algorithm. You have a tests to confirm the output, and near realtime feedback that you assumptions are correct. The rest is obvious. Mission critical component TDD, complicated refactor TDD, algorithm you need to validated TDD. Anything else write the code and get a peer review.
No amount of tests can prove that an algorithm is correct. At best, they prove that an algorithm works in a particular case.
And generally, I would say that the more interesting sorts of tests (fuzz testing, large-scale system testing) are extremely unpopular with software engineers because "they suddenly fail without reason". Not quite as unpopular as actual proof that an algorithm works, like implementing it in coq for instance, but very unpopular.
If you were to always write tests immediately after you write a few classes I don't think it would make a difference. However from my own experience I never write nearly as many tests after the fact.
Up until now it has mostly been opinions and biases and even though many popular programmers[1] have been saying this for a very long time, it's great to see a controlled study done about it.
This makes it a fact and a great counter argument for helping a lot of programmers who are being forced to practice TDD because of the generally accepted claims in productivity and code quality associated with doing it.
Keep in mind though, that the study shows that it's no better than testing after iterative development. Testing is still required, and I'd wager that the study participants following the ITL process didn't have external business pressure to skip the "test-later" bit...
Software development varies enormously. Flight avionics software differs from video game software differs from a spreadsheet differs from an order-entry system differs from laboratory analysis software differs from a web browser and so on. Flight avionics differs from a commercial jet liner to a fighter plane to a model airplane. Some projects have huge budgets and others have shoestring budgets. Some projects require extremely high reliability and quality; cost is not an issue. Other projects can be quite buggy, low quality but still useful -- cost effective.
Developers vary as well. Some temperamentally find something like TDD useful. Others do not.
There is no one software development methodology to rule them all.
331 comments
[ 3.0 ms ] story [ 270 ms ] threadIt's really a matter of code that lacks tests that's an issue and especially of code which isn't designed with testing in mind. I think they overinterpreted the value of TDD as simply test-first. Test first can be good but it's more a motivational tool and a productivity simulator (yay look at those green checkmarks!) than a real benefit.
The end result is identical, this sort of craps on it as a motivational tool too if you consider that only as a factor in development time at least. It'd be interesting to see developer satisfaction included in some way too.
My attempts to work in a TDD style ran up against decades of intgrained habits and I never really found it satisfying or natural.
That's not to say that you should try it again; I like TDD a lot myself, but software methods are so interdependent that I think each person has to judge what works best for them.
This is definitely not my experience. As Kent Beck says, TDD is a design technique. It forces you to always start thinking of the code from the outside of the unit. If I build the unit first and add tests later, it's more likely I'll end up with something where the API reflects the implementation. With test last, I'm also less likely to test everything well; after the implementation is done, I believe it works.
Will you?
Are you sure?
Do you have data to back your supposition?
My contention would be that at the end of the day, the requirements of the interface to support unit testing will result in a very similar set of design choices, whether you write those tests up-front, as-you-go, or after-the-fact.
The only difference is that if you write them after-the-fact, you may push some amount of refactoring to the end of the process instead of doing it along the way.
But I'll bet you have about as much data as I do to back your beliefs. ;)
That's my experience. I started doing TDD more than 10 years ago, and it took me about a year to fully make the switch from test-after to test-first programming. I regularly try experiments with different personal code bases.
If you're arguing from personal experience, that your code ends up just the same either way, good for you. But I suspect you're arguing from theory here.
Getting the testing as close in time to the implementation as possible probably makes a bigger difference than first vs last.
However, for me that's a hard set of behaviors to stick to. If I let myself say, "Oh, I'll test that later" I'm likely to get lost in the problem, write bunch of production code, and then say, "Well, it already works, but I guess I'll add some tests." There's no clear stopping point.
Test-first programming, on the other hand, is easier for me to stick to. If I don't have a broken test, I write one. If I have made the test pass, it's time to write another test before I write more production code.
Why would we assume TDD would improve code quality or development time?
If, at the end of the day, the result is the same volume of tests and the same level of overall coverage, the order of activity would seem to make no difference. What would lead you to believe otherwise?
The real flaw in both methodologies is that it's the developer checking their own work.
I'd make the claim that if you really wanted to do it "right", you'd have a specification, one developer writing tests to the specification, and a different developer writing the code to make the tests pass.
Of course, that's probably not workable in practice... make for an interesting experiment, though (and yes, I'm too lazy to search Google, only to discover someone else already came up with the idea).
But I know some people are wired differently; they'll think a lot more about scenarios first, then code after they have everything accounted for. For them, TDD as a philosophy seems more fitting.
I think the chasm exists between _untested_ code and code that has tests. I've never understood the seemingly-religious zealotry behind TDD as an XP practice. Just like pair programming... if it works for you and your coding style, awesome. But don't force it down my throat or act like it's the One True Path to clean code.
This applies to so many things in Software Development - text-editors, variable names, plugins, architectures, operating systems...
Edit: ED IS THE STANDARD TEXT EDITOR! [0]
[0] https://www.gnu.org/fun/jokes/ed-msg.html
Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth between the two testing methods even more rapidly.
When doing UI code (with something like React) it's next to impossible to do TDD - you can't know the structure until you actually implement it.
If I'm fixing a bug in some sort of non-ui code, I often find TDD to be very helpful. Create a test case that demonstrates that failure, then fix the implementation to make the test pass.
I'm personally a fan of "readme driven development" (https://news.ycombinator.com/item?id=1627246). It's a nice compromise between tons of upfront planning and nothing at all.
I had thought of involving users in the design of new features via the support forum but I'm afraid of how that might turn out. A readme type document would probably be about as good a way of doing it as any though.
I'm one of those people. But I find it hard to believe that a test suite can reflect everything I understand about a problem domain. Focusing too much on unit tests makes it difficult to find the general rules that govern the behavior of a system. Have too few unit tests, and they won't capture all the subtleties and corner cases. Have too many unit tests, and they won't fit in your head.
IMHO untested code isn't always a bad thing. Anything related to privacy, security, or data integrity should be heavily tested.
But beyond that, code should need to earn its tests. (At least in a web startup context.) After all, the core of agility is being able and willing to go in a different direction when something isn't working.
In addition, if you are truly in the earliest stages of your startup (pre-revenue/traction) the tests are not needed -- but you have to be _very_ aware that you are making a tradeoff for sheer velocity and you will have to pay the price later on. I've seen too many startups fail to pay the price and it comes back to haunt them and reduces their overall velocity.
"If your code doesn't have tests then you can't refactor" is pure dogma and trivial to falsify (go on, refactor some untested code now!). More likely to introduce bugs? Perhaps. But relying on that absolutist statement about refactoring will alienate me every time.
80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions.
So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).
Naturally, I never get this part right at first, and so I wind up refactoring a lot as I go along. ("Build one to throw away", I believe this has been called.)
Now in my experience, TDD requires me to refactor more during this process than tests-after-code. But suit yourself. Changing the order within the last 20% won't be earth-shattering one way or the other.
In my experience, using a type system to do this requires much less effort and refactoring.
Unit tests can demonstrate that my code is correct; it can say nothing about how my code is used by others.
In contrast, a type system can extend those protections downstream.
(Yes, yes, unless you go way off the deep end of dependent types, you still need tests. But a powerful type system can systemically prevent a huge number of very common bugs.)
As for dependent types, I'm not sure that even then you would be able to ensure that, say, your cache is correctly invalidated.
Yes. TDD came from developers who worked in weak/dynamic typed languages.
It is mandatory to discover errors, that are otherwise discovered instantly for free by compilers in the stronger typed languages.
Strangely, the TDD folks never outline this point.
I never catch all the design errors with non-coding design work - this gives me the high level view which is critical - but there are always some logical errors and bugs I've failed to model correctly which early tests catch.
But yeah, the optimal design-implememt-test probably depends probably very much on the implememter and on the problem to be solved.
The third meta-advantage is that if I'm writing hobby code for myself and the profject is large compared to the weekly time allocations available in my sparetime I get something to help me remember where I was (say a month back) and gives me something to compile and run immediately (which is a huge motivational booster for me).
Not sure if that claim is wrong or not, but I will state it anyway. If you write in TDD manner and you decide to change the API/abstraction a little bit because of something you didn't think of, you will have to correct both tests and the code. When you do test-after-code it's much easier (IMO) to hit correct abstractions and interfaces because you are working on a living thing, not some artificial test cases you have to figure out. Then you write tests to ensure those abstractions are respected and in working order.
TDD people, please correct me if I'm wrong.
EDIT: disclaimer: I'm a front-end developer working mostly in React and Redux
When I was a programming beginner and started learning about TDD, I was always thinking how soon to start using it because everyone says it's such a good idea. If you would search anything about it, you'd rarely find a resource not marvelling at its surprising benefits (at least, that is how it was few years back). Granted, it may have its merits but is it really a good idea for everything? No way!
One of the anti-patterns I have seen in programming world is how everyone jumps on adopting / advocating a 'good' practice without understanding how it fits the project. I wrote about this a while back[1]. Cue in — git flow, TDD, not using goto, DOM Parser instead of RegEx.
[1]: https://shubhamjain.co/2015/10/26/imperfect-best-practices/
Sometimes it is hard to write the tests, but it is easy to see if your code works - sometimes it is easy to write the tests, but not the code.
I would call it 'do first what is easier'.
But it is important to write the tests quickly after you wrote the piece of code - so that you don't forget something.
If you have no idea what the results should look like, then TDD is pointless.
Writing a list of tests first is quicker, though, and working through the list, updating it as I learn more, tells me that my initial ideas were wrong often enough that I'm almost convinced that the true secret of TDD isn't the tests at all. It's that checklist, and the purely mechanical process of working through it, checking them off one-by-one and fixing it up as I go along.
To relate this to your situation, I often find that I'll only get started with a fairly fuzzy idea, and the checklist will be short. I can still expand it as I go, knowing that I'm working on as solid foundations as I can given the constraints.
I think there's a danger in writing off practices like TDD with "if it works for you..." because it avoids discussion of why it works for some people and not others. That in turn gives people license to ignore it simply because it's unfamiliar, and the brain doesn't like unfamiliar things by default. That's a shame, because for some of those people they'll be stuck in local optima, writing off the one thing which could break them out of it.
I generally find it better to engage in "top-down" development - specification->tests->high level code->low level code as it's cheaper to fix mistakes at higher levels first.
If I'm doing purely experimental code though - where I'm not sure of what I even want or what the output should be, tests are just a waste of time.
It also loses its effectiveness the more declarative the code is. If I'm essentially tweaking configuration or HTML there's no point.
* write a failing test (RED) * implement minimal functionality so all tests pass (GREEN) * REFACTOR
and overcome the initial revulsion you will get a ton of benefits that you didn't know existed. I'll first give a few that are more on the code quality side
* You have a complete test suite, without redundant tests * Tests are usually not that mock-focused [] as the implementation-after-test didn't introduce random roadblockers for testing. tests do assert the relevant properties (tests-later code bases often have code bases that don't assert much or properties you don't care about). * When you write your test first, you basically write a small example how it feels like to use the API you have in mind for solving your problem. This drastically improves quality.
Some non-technical benefits of TDD
* I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing. * TDD is programming gamification. You are rewarded with regular, small successes of having added another test and made it green. So whenever you ask yourself: OMG, have I made any progress in the last 3 days, you can actually tell: Yes, I added 15 tests and made them pass - I objectively got closer to implementing that feature. * Not an issue where I work, but have often heard about this from other places: If tests are there first, no one can tell you to not spend so much time on writing tests. TDD-style is protecting your boss from shipping that car without having the brakes checked. * The same line: TDD gives explicit room for refactoring.
[*] obviously one cannot avoid mocking especially when dealing with resources external to your code base, yet I have hardly seen TDD-originated tests that mocked
This made me think. There are many axes of software development, and aside from TDD/non-TDD axis, there's also the bottom-up vs. top-down axis. By top-down, I mean you start with a skeleton that provides an interface to the outside world, and the fill in the implementation details. Bottom-up, on the other hand, means you first write some routine that you know will be vital in some form or another, but without immediately seeing how it will be hooked into the interface to the outside world.
Top-down has the advantage that you can have something runnable quickly, and it seems to me that TDD requires top-down development.
But somehow, there have been times when I was working on a piece of code and it just flowed more naturally in a bottom-up manner. It has happened that I went for a day (and very rarely longer) working on tricky algorithmic code before I even tried to compile it for the first time.
This is definitely something that I try to avoid when I can, it's just that sometimes it works out that way. And it is totally incompatible with the TDD way of doing things.
How often this happens surely depends on the type of project you're working on. As usual with these things, it seems like TDD can be a useful inspiration, but it shouldn't be taken as dogma.
I dont think writing Tests first makes you work top down or bottom up.
If you want to work bottom-up (I like to work bottom up) you just start by writing tests for your bottom up functionality.
And of course I don't want to impose TDD on others, but I would like others to try it out sincerely (recommending kent Becks book for this. Don't do it without guidance, many people who reject TDD after trying it out were doing some kind of Test-first method, but didn't get the benefit because they were not aware of stuff like get one test at a time to complete, etc.)
---------
> shouldn't be taken as dogma.
Life in software engineering becomes much more relaxed when you realize that every piece of advice and technology is in fact based on dogmas and is not based on sound empirical, statistically relevant evidence :-) (apart from the pure math part).
It is just really really hard to study software quality in an objective way that satisfies scientific criteria. So we are kind of stuck with hearsay in IT
So TDD is kind of dogmatic. So is OOP, or the rejection of gotos.
It doesn't.
All the article says is TDD has little or not impact... it does not say unit testing as a practice has no impact.
Seems to be a common misunderstanding around here that you, too, have fallen victim to...
I work with people who openly believe that unit testing is a waste of time. Personally, I think they are afraid, because they don't even know how, and don't care to learn. They would see that title, and that's all the confirmation bias they would need.
Thanks for assuming the worst about me though.
My guess is that a lot of folks got introduced to unit testing through test-driven development and, as a result, conflate the two, assuming the former necessarily implies the latter.
Speaking for myself, I was writing UTs long before the TDD fad landed and so I never picked up the habit. It's just not the way my developer mind works, and I've not found it compelling enough to try and re-train myself.
That said, not believing in the value of automated regression test (of which UTs are the lowest hanging fruit) is utter madness...
Unit testing can be a waste of time.
Perhaps because it's less stressful. You think about system design as you code, instead of only when you hit a wall and have to rewrite everything, or when you have to clean up for code review.
Either way, if it has little to no compact on dev team or code quality, I bet the positive impact TDD has on team morale would make it worthwhile.
TDD has a very specific meaning. It means you write tests, then write code that passes those tests. That specific order. If you're not doing that, you're not actually adhering to the definition of TDD.
TLD could mean, for example, writing a module, class, or function with a defined interface that implements the contracts for that interface, then writing the suite of tests to validate that module, after which you move onto the next module.
Strangely enough, you can do that while you're developing the product without adhering to the process dictated by TDD.
In contrast, TDD has been a waste of time for me for UI-based work, as the effort needed to properly expose the functionality under test is too great and the requirements and design change too quickly to be worth it.
In the latter case, writing some deterministic UI tests against mock data after the requirements and implementation have settled has proven much more effective in preventing regressions.
It also produces superior software in my experience.
So it's win win then ;)
Another approach is to partition your code into what Gary Bernhardt calls an "imperative shell" of code that touches the outside world and a "functional core" that does not. Then unit test the functional core, which shouldn't require special testing libraries, and validate that the imperative shell works by the occasional manual or "outside" test.
1) http://www.webtigerteam.com/johan/en/blog/error_driven_devel...
On the other hand, TDD for operations that involve I/O (including user interactions) were not helpful at all.
I view it more as important for breaking the growth of testing effort in an iterative project. With each release the scope of what should be tested to fully test a project climbs and unless a team wishes to linearly increase the size of its test team its all but certain tests will be skipped.
TDD gives us the ability to always full regression test as its just machine time. Its a safety factor in knowing nothing is broken which in turn gives us confidence we can refactor.
But tests for any of the interactions between subsystems is quite problematic. And then testing on the device might be problematic due to space constraints, but testing on a simulation is also problematic... I don't know how you'd realistically test it.
I'm not making a stand about TDD here - I just think we need to have much better computer engineering science studies if we want to have significant results.
I also think of TDD as a sustainability practice. If I'm writing a small thing that I do not intend to maintain, I won't bother with TDD (or with tests at all). But I'll definitely TDD something where I expect to come back to it frequently, especially when I initially don't know the requirements, and I expect requirements to change over time.
In practice, I suspect a lot of the interesting questions about software are effectively unanswerable with the budgets available to CS profs. I can't imagine really answering this question without doing something of the scope of a substantial medical study.
That's all well and fine, and possibly completely accurate. However, many people's takeaway is going to be the out-of-context & incorrect title of this post. (It does not say TDD is worthless - it says it is essentially the same as TLD)
I've always looked at TDD as a tool to help push less experienced, less "educated" developers into 1) even using tests at any point of the development cycle, 2) creating tighter, cleaner and MORE TESTABLE code by the time they've reached the end of the cycle.
So, if your team is and always will be well-educated, experienced programmers who already understand how to always do everything correctly from the beginning, feel free to use either method.
Otherwise, I'd urge you to consider TDD.
I've met a few that couldn't code let alone write proper tests, even though their foci were not policy or say the lighter side of user experience.
That's only personal experience, however, and mostly out-of-date
That said, I would hope any of them would be a step up on the new programmers from the "churn 'em out" short courses many are forced to take as the only affordable way they have to get into the industry, but then are left with little knowledge beyond how to get the requirements fulfilled.
But moreover, I think it's important to understand why Beck pushed for TDD.
TDD is like saying "I'm going to floss before I brush every time, no matter what."
But, when people don't do TDD they typically aren't all saying "I'm going to brush and floss afterwards every time, no matter what."
Instead, most say "I'll floss regularly at some point, but I don't have time now, and it takes too much effort. I'll floss here and there periodically, maybe before my monthly meeting or big date night."
Another reason Beck pushed for TDD was method and solution complexity reduction which results in lower time and cost required for maintenance because code is simpler to read and understand. Again, with ITL, you're still writing tests for everything, so you'll see those benefits. However, if you fail to write some or most tests, some developers will write overengineered solutions to things and have overly long difficult to follow methods that will make maintenance suck more resources.
If you want to go beyond this study, though, Beck, Fowler, and DHH had a critical discussion about TDD in 2014 that's worth checking out:
http://martinfowler.com/articles/is-tdd-dead/
I think it would be very hard to argue that Waterfall ~= $CURRENT_BEST_KNOWN_METHODOLOGY.
[1] Of course, this isn't usually what happens in practice when pharmaceutical companies are doing their own testing, but it's what should happen if you actually care about efficacy and not just PR/sales.
And generally, I would say that the more interesting sorts of tests (fuzz testing, large-scale system testing) are extremely unpopular with software engineers because "they suddenly fail without reason". Not quite as unpopular as actual proof that an algorithm works, like implementing it in coq for instance, but very unpopular.
Doing it at an expert level is very different from an untrained novice winging it.
The research shows no significant difference between TDD and iterative test-last (ITL) development.
Could the title be updated? To show that it is a comparison of TDD vs ITL/TLD.
FTA: In this paper we reported a replication of an experiment in which TDD was compared to a test-last approach.
Very different title.
This makes it a fact and a great counter argument for helping a lot of programmers who are being forced to practice TDD because of the generally accepted claims in productivity and code quality associated with doing it.
[1] http://david.heinemeierhansson.com/2014/tdd-is-dead-long-liv...
Developers vary as well. Some temperamentally find something like TDD useful. Others do not.
There is no one software development methodology to rule them all.