I must admit that I don't fully do TDD, but sometimes it helps me getting my tests out first, so that I can "figure out" what to do in my real code whilst running the tests.
I see the return values of certain functions and computations, and while debugging my failing test, I can see a path to the solution I need.
Once the test succeeds, it's time to clean up the ugly code and refactor it into clean subsections, be it functions or classes.
So yeah, TDD is a great tool.
One of many in my toolbox.
Me personally, I would take several projects with TDD and non TDD methodologies, control variables like team size, project size, team experience, etc. and compare outcomes like defects rates, schedules, performance, etc.
Try looking at SEED http://evidencebasedse.com/ . TDD comes out not to bad. Quality correlates very well with # of unit tests regardless of methodology. TDDers tend to have more unit tests. Therefore TDD tends to produce quality code at better rates than other methodologies because it tends to produce more tests.
Thanks for sharing, I agree with the points you made. The only thing I would add is that not everything is testable in the sense of given: input, expect: output.
A common scenario I encounter is that sometimes when you are refactoring a method and refactoring it into multiple methods (for better readability or segregation); that you are required to make the new (sub) methods publicly available for testing purposes. This goes against the intention of improving readability because now you have to expose certain methods that you would rather haven't be called independently. In such way, TDD helps identify artifacts with too many responsibilities.
The best tip I can think of to give someone approaching/doing TTD is to make a check-list beforehand of what you are trying to achieve. This helps to focus your attention on usage patterns and identify 'end-points' (end points being testable artifacts). Just because you're doing TDD, doesn't mean you don't have to have a plan to start with.
When talking about this common scenario, there are two ways of think. Some people argue that you shouldn't expose private methods only to test them. You should do it through the public method to keep the encapsulation. Others argue that yes, you should expose these methods to make easier to write tests and to identify possible bugs.
It depends, we have to analyze each case and choose the advantages and disadvantages.
In Java you don't have to make the methods public, you can just make them default scope. The test cases can see them because they are in the same package but the rest of the system can't.
I'd imagine this is similar to "internal" in C#. If so, that still doesn't equate to being private, as other objects can see them within the package/assembly. To the outside world this is fine, of course, however it removes that additional organizational capability within the package.
I've often found that not being able to test something in a sense of "given: input, expect: output" is a smell that you're taking the wrong approach and that a better factoring for your code exists (this obviously isn't always the case, perhaps the most common exception being testing integration with third-party code).
There's nothing requiring you to specifically test every function in a class - though, as others have said, you can set the access level to default or protected to have package-level access for tests.
Strictly speaking, though, refactoring should never require new tests, unless you're specifically refactoring to increase testability. New tests should be related to specific bugs or feature requests.
How many of the <fixnum> benefits of TDD and unit tests are achievable through the use of sufficiently advanced type systems? Disclaimer: I'm not using advanced type systems to replace unit tests myself, but the more I read about them the more I suspect that most of the tests I've ever worked with (a small sample to be sure) could be obviated by pushing the logic they test for into structured types.
This is one thing that has irked me a little bit as I've gotten more involved with TDD and unit testing in general: the logic in many of my unit tests is near duplicate logic that is implemented within my application.
My thought is that the general advantage of TDD is that you are thinking about your eventual implementation from the outside. You write tests in a different mindset based on expectations, rather than when you are implementing the production code where you are more concerned with the details of logic, etc. Unit tests also tend to be a bit more readable than the actual implementation logic, and so act as a nice form of documentation as a side-effect. In this way, test-first development provides a more gradual transition between a specification or design and the end implementation, which results in fewer missed requirements, allows for easy refactoring, etc.
This is of course only related to TDD however, not unit testing in general. Hence, it still irks me that many of my tests look so similar to the logic they are testing, and thus aren't really testing for correctness at all.
Think of unit testing as auditing the contract between two pieces of code right at the interface by replacing one of the two pieces of code (usually the client or caller but you can replace the code that provides the response just as easily) by a program that exercises the other layer.
Like that you build your confidence layer by layer and with every set of tests you verify that you are still building on solid ground instead of quicksand.
That's why you test at the interface level, it allows you to swap out the implementation without having to re-work all the test code.
I went through a phase of this when I started writing automated tests. It occurred when I started using a mocking library for the first time. The library was so advanced I could basically test my implementation instead of my api. EG: "first you call foo.bar(), then you call foo.baz()" As a result, my test and my system under test looked very similar.
I'm not sure if you're in the same situation I was in, but let me tell you: My situation was an anti-pattern. All I was testing was that my code does what it does. I wasn't testing that my code was working.
Make sure you're not making the same mistake. If you are, a symptom would be that lots of tests fail when you're only refactoring.
Not quite, but I get what you're saying for sure. I think jacquesm said it well in his reply to me that you're more testing the "contracts between two pieces of code right at the interface", and it shouldn't be any deeper than that to avoid what you're describing.
The symptom of many tests failing during refactoring is something I'll be keeping in mind, however.
Some issues are removed with a type system and wouldn't require writing tests to verfiy (the compiler would be doing this for you). I wouldn't take that fact as being in favor of using a type system, but more of a reminder that using a dynamic language without tests is reckless.
I just want to note that with an advanced type system, "type testing" can cover a lot more than people seem to give it credit for. An example I saw someone give on here the other day was that he had the types for his red black tree set up so that code that produced an unbalanced tree would never compile.
https://news.ycombinator.com/item?id=5987527 I'm just getting into Haskell myself and I can't really explain those 10 lines myself (I also haven't implemented a red black tree).
I watched a talk by Yaron Minksy the other day that also talked about the idea that you should "make illegal states unrepresentable" that was really good: http://vimeo.com/14313378 (he starts talking about that at around ~17:50)
First we create two differently typed tags for Red and Black.
data Red
data Black
Note this differs from something like
data Color = Red | Black
in that Red and Black are distinct at the type level---the compiler will actually complain if one is used where the other ought to be.
Now we state that Trees must have Black roots. This is done using a GADT (Generalized Algebraic Data Type) which allows us to build constructors of types which restrict some of the types.
data Tree a where
Tree :: Node Black n a -> Tree a
For instance, compare PoorTree which would let us make Trees with Red roots
data PoorTree c n a = PoorTree (Node c n a)
It really does nothing more than wrap up a Node to hide it and change the interface (if you want).
Now we need one last piece before digging in to the core. This is the existence of Type-level Naturals. The idea is that, like Red and Black, we introduce two new distinct types, one basic and one parametric.
data Zero = Zero
data Succ a = Succ a
which lets us write (very verbose) naturals like (Succ (Succ (Succ Zero))) == 3 and have the type reflect this exact value
(Note this is a little weird, since we can also form the value (Succ 'a' :: Succ Char) which doesn't make much sense. More powerful dependently typed features help here.)
Now consider just a branch in the tree, we form that with a simple data type
data NodeH l r n a = NodeH (Node l n a) a (Node r n a)
which tracks the left and right colors in its type using the l and r type parameters. It also guarantees the constraint that both the left and right child trees have the same black-height. So, if (black3 :: Node Black (Succ Zero) Char) then (NodeH black3 'a' black3 :: NodeH Black Black (Succ Zero) Char). And if (black2 :: Node Black Zero Char) then (NodeH black3 'a' black2) is a compile time type error.
So with all these pieces, we can encode the Red/Black criteria directly into our Nodes. Remember already that if we have a Tree it must have the first Node be Black. We can now construct new Nodes in three ways, each of these ways being constrained so as to enforce the balance rules.
data Node t n a where
-- If we construct a Node with Nil, it forms a leaf.
-- Not only are we not allowed to associate left and
-- right branches (we don't get to give it a NodeH type),
-- we're forced to set the numerical type parameter to Zero ~= 0.
Nil :: Node Black Zero a
-- If we construct a Node with BlackNode, we can pass a
-- subtree with the NodeH parameter, but the type of node we
-- get back is necessarily going to increase its height
-- since (Succ n) is like (n + 1).
BlackNode :: NodeH t0 t1 n a -> Node Black (Succ n) a
-- Or we can construct a Red node. In this case, our black-height
-- remains unchanged, but we can ONLY do so when we've got a NodeH
-- branch with two black subNodes.
RedNode :: NodeH Black Black n a -> Node Red n a
So, it's a type error to construct a tree in this format which doesn't obey the Red/Black balance rules. Any transformations you write on the tree will be rejected by the compiler if they break the rules.
Great explanation. I would have killed to see something like this back when I had no clue what GADTs were about. The only thing I would add is pointing out that the syntax you used in the end kind of does the same thing as
data Node = Nil | BlackNode | RedNode
but with the extra extra restrictions on the type variables.
Thanks, and true. GADTs are weird creatures when you first see them. I originally thought it was entirely a syntactic thing!
As weird as it is, I always recommend Agda/Coq for learning GADTs. It feels like overkill, but Haskell's DT stuff is really wonky, so it's nice to see GADTs expressed in a clean manner before trying them out in Haskell.
It's not type "testing", it's using a strong, static type checking system before compilation as a testing measure. I.e., you never have to write code that asks about types, an automated program simply ensures that the types that your program expresses "make sense together".
This is often conflated with type inference where the program code itself is analyzed in order to compute what type it "ought to be". Some combination of type checking and type inference is a powerful, fast, comprehensive tool for encoding how your program ought to behave alongside the program itself.
Even with advanced type systems, the sweet spot will be a mix of types and tests. With types, you can prove many claims, including some you can't test for. With tests, you can gather evidence to support the claims you can't easily prove with types. So you need both.
Elaboration:
With advanced type systems you can prove that your code satisfies (or fails to satisfy) any property that you can encode as types. The more expressive the type system, the more properties you can encode, and the more kinds of claims you can prove.
But there will always be some properties that cannot be proved or that are too expensive to encode within a given type system. If you care about these properties, you'll have to find some other way of making sure they hold. So, even with advanced type systems, you'll end up writing tests.
The flip side is that there are many important properties that you cannot adequately test for (e.g., "my code is free of injection vulnerabilities"). If you're not using proofs to make sure that these properties hold, there's little reason to believe that they do hold, regardless of how much testing you do. So, even with lots of tests, you'll end up using types.
I find the decision tree tends to start with whether or not I even understand the problem well enough to generate constraints. After that there are serious questions (today) on whether it's worth formalization. After that, you're probably knee-deep in discovering the difficulty of any proof obligations you've given yourself. Then, finally, you can answer questions about how pervasive the encoding difficulty might be.
If those fail at any point it drives you to either use a more permissive type or to just test some useful cases. Not infrequently, test cases can be upgraded into more powerful types.
If you test within a small enough scope (even at runtime and without a type system; eg. input to each function, religiously) then you don't really have the "do I understand the problem well enough" architectural issue, which in any event is not something you can test for ;)
I disagree completely. Just because you've made an interface doesn't mean it's a good one nor that it will solve your problem. You may also not understand how that interface interacts very thoroughly. Most of programming is figuring out the right interfaces.
It's one thing to name a few example inputs and outputs of a single function, it's another thing to think about how a single function interacts with itself (such as what's needed to write a quickcheck property like "for all strings, `reverse . reverse == id`"), it's yet another thing to think about how collections of related functions interact (like seeing some Category laws), and it's a whole set of further things to see all that thoroughly enough to encode it in a logic system.
Err, I'm not saying understanding the problem is irrelevant. What I'm saying is that it's not an issue that you need to be overly conscious of in testing since it's not something you can test for and properly loosely-coupled code will not exhibit dependencies on architectural assumption.
UnitTests != TDD. Why is this confused so often? TDD is an optional variant to write tests. The benefits of TDD are questionable. The benefits of UnitTests are beyond question.
You seriously blame a vaguely defined, unorganized group of people of deliberate conspiracy?
A much simpler and less nasty explanation could be that to people who do TDD, the difference between 'doing TDD' and 'writing good unit tests' is completely uninteresting. It's all part of the same engineering practice.
This blog post clearly seems to target people with an interest in TDD, so I don't see what's wrong with that.
Actually, it's quite the opposite. TDD advocates and researchers have been trying to clarify the distinction for years. One of the motivating factors of BDD was to rebrand TDD with less of a focus on the word "test," because that word causes a lot of confusion among practitioners.
Also, please drop the word "cultists." You can disagree with a technique on legitimate grounds without resorting to ad hominems. I'm downvoting you because you're not adding any substance to the debate.
Your article confuses the two constantly. You can't use TDD without using unit tests, TDD is a way of using unit tests. You can use unit tests without TDD. Most of what you talked about had nothing to do with TDD and was just about unit tests.
I wrote about using unit tests with TDD. You can use unit tests without TDD. Of course, some of the TDD benefits are also achievable when we use only unit tests.
TDD has no benefits, that is why the confusion is such an issue. People keep pretending the benefits of testing are benefits of TDD, when TDD is entirely unnecessary.
Can you cite the research on that? I've read much of the research on TDD, and the results are inconclusive largely because they're inconsistent. At best, there is some weak evidence of some actual benefits but at some small cost. The two major problems that I see with the research in TDD are (1) there's little confidence that TDD was applied correctly in many of the studies, and (2) the studies have too low of statistical power to make conclusions.
I think you're relying on a misunderstanding of TDD to make your conclusion. You say that "people keep pretending the benefits of testing are benefits of TDD." While it's true that many practitioners confuse the two, that is not the purported benefits of TDD. It's not fair to draw a conclusion based on that premise.
If you've read it, why do I need to re-show it to you? There is no evidence that TDD provides any benefit. There is evidence that testing provides benefits. TDD salesmen offer studies showing the benefits of testing as proof that TDD works: http://haacked.com/archive/2008/01/22/research-supports-the-...
> There is no evidence that TDD provides any benefit.
You originally said, "TDD has no benefits," which is a definitive claim and not supported by evidence. That's vastly different from saying, "There is no evidence that TDD provides any benefit." You seem to be conflating being unable to draw a conclusion with drawing a negative conclusion. They're not the same thing.
> If you've read it, why do I need to re-show it to you?
Because the research I've read doesn't support your conclusion that "TDD has no benefits." I'd like to know if there's a body of research out there that I haven't read yet.
You seem to have an axe to grind with TDD salesmen. That's fair enough. Salesmen aren't in the business of advancing knowledge. They're in the business of persuasion. If you want to criticize salesmen for not being scientifically honest, be my guest. Sometimes, this tactic causes harm. For example, if a salesmen persuades someone to buy homoeopathic remedies instead of traditional medicine, people could die.
I think it's good that you're skeptical of the salesmen, because they're business doesn't require them to be scientifically honest. But that shouldn't inform your opinion of TDD (or any other subject), because there are others in the world whose business does require them to be scientifically honest. And they're the ones whose words you should read to form your opinion. And that is why I asked for sources.
>You originally said, "TDD has no benefits," which is a definitive claim and not supported by evidence
No, you don't have to prove a negative. If it has not been proven, then it does not exist. Until TDD has proven benefits, it is correct to state that it does not have benefits. Just as until unicorns are proven to exist, it is correct to state that unicorns do not exist.
>Because the research I've read doesn't support your conclusion that "TDD has no benefits."
Yes it does, you just want to pretend that proof needs to work in reverse. It doesn't.
>But that shouldn't inform your opinion of TDD (or any other subject), because there are others in the world whose business does require them to be scientifically honest. And they're the ones whose words you should read to form your opinion
My opinion is based on the evidence, that's the point.
That's equivalent to saying that a scientific claim doesn't have to be falsifiable. That's not true. A claim has to be falsifiable for a reason.
> If it has not been proven, then it does not exist.
So the Higgs Boson didn't exist six months ago? It wasn't proven, because there wasn't enough evidence for a statistically relevant result until March 2013. So does that mean that prior to March, you would have concluded that the Higgs Boson doesn't exist?
If I had to sum up my problem with your reasoning in one sentence, it would be "belief is scalar, not boolean." Evidence should adjust your belief state somewhere between 0 and 1, but it should almost never be exactly 0 or 1.
> Just as until unicorns are proven to exist, it is correct to state that unicorns do not exist.
That's a variant of the "existence of god" problem. It's a non-falsifiable hypothesis. That doesn't fall within the realm of science, because it's not subject to empirical evidence.
> My opinion is based on the evidence.
I still haven't seen the evidence to support your conclusion.
TDD's claimed benefits are falsifiable. Much like the Higgs Boson prior to March 2013, they have been neither proven nor falsified. But that doesn't mean we can't form a tentative belief based on the weight of the current evidence available.
The weight of the evidence I've seen leans toward indicating some real benefits beyond normal unit testing. It's inconclusive because the studies lack statistical power and procedural consistency, but that doesn't mean we should automatically jump to a hardline decision in either direction (positive or negative). (Again, belief isn't boolean.)
>That's equivalent to saying that a scientific claim doesn't have to be falsifiable
No it isn't. Simply stating the opposite of reality doesn't make it so.
>It wasn't proven, because there wasn't enough evidence for a statistically relevant result until March 2013.
That is up to interpretation. Many felt there was enough evidence. This is not a case like that, there is literally no evidence at all for TDD. None. And you are demanding that I prove it doesn't have benefits. Prove programming while wearing pink socks doesn't have benefits.
>I still haven't seen the evidence to support your conclusion.
That is your fault, not mine.
>TDD's claimed benefits are falsifiable
And have been falsified, like in the link I provided you. But TDD salesmen use that falsification as evidence supporting it.
You can't say there's no evidence and then turn around to draw a conclusion two seconds later.
> like in the link I provided you
That is but one blog post. Science doesn't work by cherry-picking results. That's a tactic used by climate change deniers and homoeopathic medicine supporters. The evidence must be taken in full.
In addition, I skimmed the paper cited in that article [1]. It appears to be measuring TDD's effects on external quality. That is not a purported benefit of TDD. Most of the research I've seen agrees with that paper: TDD doesn't improve external quality (defined by passing acceptance tests). The contention is over whether it improves internal quality. Also note that the paper cites "process conformance" as a threat to validity, which is what I've been saying is one of the major problems with TDD research all along [2]. The experiment also used students, which is known to be a problem when generalizing results. With TDD in particular, some research indicates that it's more effective with mature developers and ineffective with inexperienced developers. If you've read the rest of the research instead of cherry-picking, you would have known that.
I'm not trying to make a bold claim about TDD. The evidence doesn't support that. I'm trying to help people like you look at the subject objectively. You can't form an objective opinion without evidence (which you claim) or while cherry-picking evidence (which you've done). But frankly, I don't think you want my help, so the last word is yours if you want it.
[2] That was my first bullet point in my first comment on this thread: "there's little confidence that TDD was applied correctly in many of the studies."
You are trolling right? There is no evidence to support the claims made about TDD (that it helps in any way). The claim that it helps has been falsified, by people doing TDD and non-TDD and there being no difference. These things are in no way contradictory, and I am no longer able to suspend disbelief and pretend you are actually trying to make a good-faith attempt at a real conversation here.
it’s almost impossible and you will lose too much time
but cover the majority of your code is perfectly
achievable. A good rule is test
*everything that can possibly break.*
58 comments
[ 3.5 ms ] story [ 111 ms ] threadI see the return values of certain functions and computations, and while debugging my failing test, I can see a path to the solution I need.
Once the test succeeds, it's time to clean up the ugly code and refactor it into clean subsections, be it functions or classes.
So yeah, TDD is a great tool. One of many in my toolbox.
That's also true for vanilla regression tests. There's nothing special that TDD adds in this regard.
Take a loot at http://www.ime.usp.br/~aniche/files/tdd-and-design-draft.pdf
What are we supposed to be looking at there? Like most TDD literature I see a lot of assertions, but no evidence to back it up.
Try looking at SEED http://evidencebasedse.com/ . TDD comes out not to bad. Quality correlates very well with # of unit tests regardless of methodology. TDDers tend to have more unit tests. Therefore TDD tends to produce quality code at better rates than other methodologies because it tends to produce more tests.
A common scenario I encounter is that sometimes when you are refactoring a method and refactoring it into multiple methods (for better readability or segregation); that you are required to make the new (sub) methods publicly available for testing purposes. This goes against the intention of improving readability because now you have to expose certain methods that you would rather haven't be called independently. In such way, TDD helps identify artifacts with too many responsibilities.
The best tip I can think of to give someone approaching/doing TTD is to make a check-list beforehand of what you are trying to achieve. This helps to focus your attention on usage patterns and identify 'end-points' (end points being testable artifacts). Just because you're doing TDD, doesn't mean you don't have to have a plan to start with.
It depends, we have to analyze each case and choose the advantages and disadvantages.
Strictly speaking, though, refactoring should never require new tests, unless you're specifically refactoring to increase testability. New tests should be related to specific bugs or feature requests.
My thought is that the general advantage of TDD is that you are thinking about your eventual implementation from the outside. You write tests in a different mindset based on expectations, rather than when you are implementing the production code where you are more concerned with the details of logic, etc. Unit tests also tend to be a bit more readable than the actual implementation logic, and so act as a nice form of documentation as a side-effect. In this way, test-first development provides a more gradual transition between a specification or design and the end implementation, which results in fewer missed requirements, allows for easy refactoring, etc.
This is of course only related to TDD however, not unit testing in general. Hence, it still irks me that many of my tests look so similar to the logic they are testing, and thus aren't really testing for correctness at all.
Like that you build your confidence layer by layer and with every set of tests you verify that you are still building on solid ground instead of quicksand.
That's why you test at the interface level, it allows you to swap out the implementation without having to re-work all the test code.
I'm not sure if you're in the same situation I was in, but let me tell you: My situation was an anti-pattern. All I was testing was that my code does what it does. I wasn't testing that my code was working.
Make sure you're not making the same mistake. If you are, a symptom would be that lots of tests fail when you're only refactoring.
The symptom of many tests failing during refactoring is something I'll be keeping in mind, however.
I watched a talk by Yaron Minksy the other day that also talked about the idea that you should "make illegal states unrepresentable" that was really good: http://vimeo.com/14313378 (he starts talking about that at around ~17:50)
First we create two differently typed tags for Red and Black.
Note this differs from something like in that Red and Black are distinct at the type level---the compiler will actually complain if one is used where the other ought to be.Now we state that Trees must have Black roots. This is done using a GADT (Generalized Algebraic Data Type) which allows us to build constructors of types which restrict some of the types.
For instance, compare PoorTree which would let us make Trees with Red roots It really does nothing more than wrap up a Node to hide it and change the interface (if you want).Now we need one last piece before digging in to the core. This is the existence of Type-level Naturals. The idea is that, like Red and Black, we introduce two new distinct types, one basic and one parametric.
which lets us write (very verbose) naturals like (Succ (Succ (Succ Zero))) == 3 and have the type reflect this exact value (Note this is a little weird, since we can also form the value (Succ 'a' :: Succ Char) which doesn't make much sense. More powerful dependently typed features help here.)Now consider just a branch in the tree, we form that with a simple data type
which tracks the left and right colors in its type using the l and r type parameters. It also guarantees the constraint that both the left and right child trees have the same black-height. So, if (black3 :: Node Black (Succ Zero) Char) then (NodeH black3 'a' black3 :: NodeH Black Black (Succ Zero) Char). And if (black2 :: Node Black Zero Char) then (NodeH black3 'a' black2) is a compile time type error.So with all these pieces, we can encode the Red/Black criteria directly into our Nodes. Remember already that if we have a Tree it must have the first Node be Black. We can now construct new Nodes in three ways, each of these ways being constrained so as to enforce the balance rules.
So, it's a type error to construct a tree in this format which doesn't obey the Red/Black balance rules. Any transformations you write on the tree will be rejected by the compiler if they break the rules.As weird as it is, I always recommend Agda/Coq for learning GADTs. It feels like overkill, but Haskell's DT stuff is really wonky, so it's nice to see GADTs expressed in a clean manner before trying them out in Haskell.
This is often conflated with type inference where the program code itself is analyzed in order to compute what type it "ought to be". Some combination of type checking and type inference is a powerful, fast, comprehensive tool for encoding how your program ought to behave alongside the program itself.
Elaboration:
With advanced type systems you can prove that your code satisfies (or fails to satisfy) any property that you can encode as types. The more expressive the type system, the more properties you can encode, and the more kinds of claims you can prove.
But there will always be some properties that cannot be proved or that are too expensive to encode within a given type system. If you care about these properties, you'll have to find some other way of making sure they hold. So, even with advanced type systems, you'll end up writing tests.
The flip side is that there are many important properties that you cannot adequately test for (e.g., "my code is free of injection vulnerabilities"). If you're not using proofs to make sure that these properties hold, there's little reason to believe that they do hold, regardless of how much testing you do. So, even with lots of tests, you'll end up using types.
You need both.
If those fail at any point it drives you to either use a more permissive type or to just test some useful cases. Not infrequently, test cases can be upgraded into more powerful types.
It's one thing to name a few example inputs and outputs of a single function, it's another thing to think about how a single function interacts with itself (such as what's needed to write a quickcheck property like "for all strings, `reverse . reverse == id`"), it's yet another thing to think about how collections of related functions interact (like seeing some Category laws), and it's a whole set of further things to see all that thoroughly enough to encode it in a logic system.
A much simpler and less nasty explanation could be that to people who do TDD, the difference between 'doing TDD' and 'writing good unit tests' is completely uninteresting. It's all part of the same engineering practice.
This blog post clearly seems to target people with an interest in TDD, so I don't see what's wrong with that.
Also, please drop the word "cultists." You can disagree with a technique on legitimate grounds without resorting to ad hominems. I'm downvoting you because you're not adding any substance to the debate.
I think you're relying on a misunderstanding of TDD to make your conclusion. You say that "people keep pretending the benefits of testing are benefits of TDD." While it's true that many practitioners confuse the two, that is not the purported benefits of TDD. It's not fair to draw a conclusion based on that premise.
You originally said, "TDD has no benefits," which is a definitive claim and not supported by evidence. That's vastly different from saying, "There is no evidence that TDD provides any benefit." You seem to be conflating being unable to draw a conclusion with drawing a negative conclusion. They're not the same thing.
> If you've read it, why do I need to re-show it to you?
Because the research I've read doesn't support your conclusion that "TDD has no benefits." I'd like to know if there's a body of research out there that I haven't read yet.
You seem to have an axe to grind with TDD salesmen. That's fair enough. Salesmen aren't in the business of advancing knowledge. They're in the business of persuasion. If you want to criticize salesmen for not being scientifically honest, be my guest. Sometimes, this tactic causes harm. For example, if a salesmen persuades someone to buy homoeopathic remedies instead of traditional medicine, people could die.
I think it's good that you're skeptical of the salesmen, because they're business doesn't require them to be scientifically honest. But that shouldn't inform your opinion of TDD (or any other subject), because there are others in the world whose business does require them to be scientifically honest. And they're the ones whose words you should read to form your opinion. And that is why I asked for sources.
No, you don't have to prove a negative. If it has not been proven, then it does not exist. Until TDD has proven benefits, it is correct to state that it does not have benefits. Just as until unicorns are proven to exist, it is correct to state that unicorns do not exist.
>Because the research I've read doesn't support your conclusion that "TDD has no benefits."
Yes it does, you just want to pretend that proof needs to work in reverse. It doesn't.
>But that shouldn't inform your opinion of TDD (or any other subject), because there are others in the world whose business does require them to be scientifically honest. And they're the ones whose words you should read to form your opinion
My opinion is based on the evidence, that's the point.
That's equivalent to saying that a scientific claim doesn't have to be falsifiable. That's not true. A claim has to be falsifiable for a reason.
> If it has not been proven, then it does not exist.
So the Higgs Boson didn't exist six months ago? It wasn't proven, because there wasn't enough evidence for a statistically relevant result until March 2013. So does that mean that prior to March, you would have concluded that the Higgs Boson doesn't exist?
If I had to sum up my problem with your reasoning in one sentence, it would be "belief is scalar, not boolean." Evidence should adjust your belief state somewhere between 0 and 1, but it should almost never be exactly 0 or 1.
> Just as until unicorns are proven to exist, it is correct to state that unicorns do not exist.
That's a variant of the "existence of god" problem. It's a non-falsifiable hypothesis. That doesn't fall within the realm of science, because it's not subject to empirical evidence.
> My opinion is based on the evidence.
I still haven't seen the evidence to support your conclusion.
TDD's claimed benefits are falsifiable. Much like the Higgs Boson prior to March 2013, they have been neither proven nor falsified. But that doesn't mean we can't form a tentative belief based on the weight of the current evidence available.
The weight of the evidence I've seen leans toward indicating some real benefits beyond normal unit testing. It's inconclusive because the studies lack statistical power and procedural consistency, but that doesn't mean we should automatically jump to a hardline decision in either direction (positive or negative). (Again, belief isn't boolean.)
No it isn't. Simply stating the opposite of reality doesn't make it so.
>It wasn't proven, because there wasn't enough evidence for a statistically relevant result until March 2013.
That is up to interpretation. Many felt there was enough evidence. This is not a case like that, there is literally no evidence at all for TDD. None. And you are demanding that I prove it doesn't have benefits. Prove programming while wearing pink socks doesn't have benefits.
>I still haven't seen the evidence to support your conclusion.
That is your fault, not mine.
>TDD's claimed benefits are falsifiable
And have been falsified, like in the link I provided you. But TDD salesmen use that falsification as evidence supporting it.
> there is literally no evidence at all for TDD
> And have been falsified
Falsification requires evidence.
You can't say there's no evidence and then turn around to draw a conclusion two seconds later.
> like in the link I provided you
That is but one blog post. Science doesn't work by cherry-picking results. That's a tactic used by climate change deniers and homoeopathic medicine supporters. The evidence must be taken in full.
In addition, I skimmed the paper cited in that article [1]. It appears to be measuring TDD's effects on external quality. That is not a purported benefit of TDD. Most of the research I've seen agrees with that paper: TDD doesn't improve external quality (defined by passing acceptance tests). The contention is over whether it improves internal quality. Also note that the paper cites "process conformance" as a threat to validity, which is what I've been saying is one of the major problems with TDD research all along [2]. The experiment also used students, which is known to be a problem when generalizing results. With TDD in particular, some research indicates that it's more effective with mature developers and ineffective with inexperienced developers. If you've read the rest of the research instead of cherry-picking, you would have known that.
I'm not trying to make a bold claim about TDD. The evidence doesn't support that. I'm trying to help people like you look at the subject objectively. You can't form an objective opinion without evidence (which you claim) or while cherry-picking evidence (which you've done). But frankly, I don't think you want my help, so the last word is yours if you want it.
[1] https://ieeexplore.ieee.org/application/mdl/mdlconfirmation....
[2] That was my first bullet point in my first comment on this thread: "there's little confidence that TDD was applied correctly in many of the studies."
[disclaimer: I run a company that does hosted CI: https://circleci.com]
But yes, I think that Continuous Integration is a good practice.