> I’m sure that TDD purists would say that I’m not doing it right so I’m not getting the real benefits of TDD.
"You are not doing it right! Take this 3-day course for a grand and buy these books. Also hire a TDD/Scrum/Agile coach for your team for a few months. There you go! (in Eric Cartman's voice)."
"...because I think it encourages conservatism in programming, it encourages you to make design decisions that can be tested rather than the right decisions for the program users.." - Couldn't agree more with this.
From the article: Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests.
Interesting. I've often found that the lack of tests leaves me absolutely terrified of making changes to large Ruby and JavaScript applications. I mean, if a test breaks, then I get a nice message right on the spot. But if I don't have tests, then the application itself breaks, and I may not find out until it's been put into production on a high-volume site or shipped to users.
Once an application crosses, say, 25,000 lines of code, it's hard to keep an entire program in my head, especially in a dynamic language and with multiple authors working on the code base. Under these conditions, large scale refactorings or framework upgrades can cause massive test failures, but the only alternative is to cause massive, unknown breakage.
Perhaps the author is more disciplined than me, but I'd have an extremely hard time actually following through and writing tests after-the-fact. After the code is "done", it's hard to keep working on it—no matter how much the "extra" work is needed.
I find tests to be extremely helpful for designing APIs. I write tests to enforce the contract my API is making with its consumers. Therefore, TDD forces me to consider the API from the user's perspective, rather than designing an API based on implementation details. That also helps to cut down on the cost of refactoring—unless I'm making an API-incompatible change, my tests should keep passing. This gives me more confidence as I'm refactoring, allowing me to make more sweeping changes.
This is a great way to create APIs that are hard or nearly impossible to implement properly or efficiently. I can imagine how this could easily lead to horrible API design like a well-known stdlib `gets` function. From the user perspective, gets is a perfectly fine function. Unfortunately there is no way you could write a safe implementation, because in C you have no way of checking arbitrary buffer size. Low level implementation details do affect how we design APIs.
That works well when your API is small and understood. But when you're exporting someone else's results (e.g., returning the result of a floating point operation,) your tests could start to become so exacting that they are more difficult to write and less reliable than the code you're testing itself. In that case, writing tests is basically writing an ad-hoc waterfall-style software specification.
I tend to write tests when I'm not confident in the implementation or when the tests are trivial to write. At least, that's how I do it for new developments. If you're releasing code that people are relying on, you should have a whole suite of user tests and regression tests.
It's not a good argument against testing, I agree. Automated testing is a huge boon for software quality (though by no means a panacea).
But TDD, in my experience, gives a slightly different dynamic. Because you generated the code being motivated by the tests, there are a lot of unit tests that don't test functional units - tests that are essentially testing implementation decisions.
I've seen situations where that big refactoring would involve scrapping or rewriting tests, and that causes its own kind of architectural conservatism.
I ended up in a compromise position. I use a lot of tests, but I only do TDD in niche cases where it suits the problem I'm solving.
> Because you generated the code being motivated by the tests, there are a lot of unit tests that dot test functional units - tests that are essentially testing implementation decisions.
I've seen this happen to myself numerous times as well, but I don't think this is good argument against TDD.
If a tests fails because the implementation details change, that tells us that we wrote a bad test, not that TDD is bad. As I get better at writing tests, I am better able to catch these kinds of errors ahead of time. There are a ton of little techniques you can use and questions you can ask yourself about your code that can make your tests better.
Furthermore, test code is just as important as production code, and we should be just as ruthless in refactoring and peer-reviewing each others tests (within reason).
TDD is NOT suitable for every problem, this is true. But I think too many people give up on it because they see "the problems of TDD" which are actually just "the problems of writing bad tests".
Note: I'm not trying to target you sago, but merely making a general observation. I'm talking more about people who don't write tests even close to the time of writing the code, or don't write any tests at all because the their tests kept breaking for no good reason and were slowing them down.
> that tells us that we wrote a bad test, not that TDD is bad
I strongly disagree. The only thing that matters to how software works for an end user is its boundary. You simply can't do TDD by only testing the boundary.
You have to write a test before each unit of code, so you have to test those internal interfaces, the way units of code work. But that is exactly the stuff that should be allowed to change. It's a mistake to assume 'implementation details' only exist within a function. That stuff you can avoid testing, but the behavior of the function is itself an implementation detail for whatever is using that function.
I simply cannot imagine TDD for anything other than a toy example where you don't end up writing tests that depend on the software's implementation, and are therefore inertia to changing that.
> we should be just as ruthless in refactoring and peer-reviewing each others tests
I strongly agree. But this is rarely part of the TDD ethos. Saying "I refactored the code and threw away 100 tests." is likely to send TDD folks twitching, in my experience.
Changing tests is work, yes. Unit tests are precisely about verifying implementations, and maintaining them is the price you pay for having these layers of verification. It's inertia in the way a safety net is a barrier.
I have always found the 'safety net' analogy uninspiring, and people who think they're tough and manly don't need safety nets, thankyouverymuch.
Now, safety equipment in a race car is something else entirely. Half of it keeps you from dying immediately on impact, sure, but the other half actually lets you go faster. The 5 point harness, for example, helps keep you from losing control of the vehicle during high G maneuvers, by keeping your torso in the seat and therefore your legs and arms in (roughly) the correct orientation to the steering wheel and the pedals.
> The only thing that matters to how software works for an end user is its boundary.
You're ignoring the fact that tests aren't only to validate "end user functionality". They're also there to make life easier for the developer; to make both initial development and maintenance easier.
If I need a custom sort algorithm for my product, I'm going to write unit tests for it's implementation. I'm also going write functional tests to make sure the end user is seeing things sorted. However, those tests serve different purposes.
The unit tests allow me to write clean, simple tests that show exactly what the implementation is supposed to be doing for different inputs. That allows me to change the implementation of the sorting algorithm later and worry less about breaking the functionality.
The functional tests are (generally) not going to be as clean and obvious, nor cover every possible partition of inputs that the implementation could get.
> If I need a custom sort algorithm for my product, I'm going to write unit tests for it's implementation.
I would start with writing an assertion about the end state of the sort. Then using syntactic substitution of the predicate calculi derive the program by assertions.
I'd then have a specification of the algorithm as a mathematical model from which I can check every possible execution for the entire domain on the sort. If I'm so inclined I would have a good basis for a proof.
The unit tests then become assertions on the implementation and don't have to verify correctness.
It would be awesome though if I could just generate the code directly from the specification!
You would? Can you demonstrate how that would work for, say, a sort algorithm with a tunable "fuzziness" parameter? Informally, after the algorithm is complete, each element is within k indices of where it would be after a traditional sort. So k=0 could just be quicksort, but as k approaches the length of the input we have to do less and less work.
It's easy to dream up algorithms and traditional unit tests for this (and easy to see how the large input space makes the approach suboptimal). I'd love to see how your approach would work.
Would you actually do this, or do you just like to imagine a world where people do this?
I'm familiar with Coq and Prolog(one of which I think you're hinting at), but I'd reach for Perl or C over either one if I needed to write a trading program or generate a statement from my accounting software or something.
I learning to use these techniques in my everyday work (cloud software).
I think of it like this: nobody is going to care if I screw up the script to balance my accounts. If I write some software that messes up the balance of everyone's account in some bank then someone is going to care.
I don't think formal methods and rigor are the sole domain of government institutions, avionics control software, and industrial robotics. The public relies on the cloud and the applications that run on it and they have no recourse if their data is exposed or disappears.
Imagine if instead of crossing over a bridge you were stopped at a booth and asked to sign a wager stating that nobody is liable if the bridge collapses underneath you while you cross it. Rich or poor, regardless of how much you pay in taxes, you should be able to rely on that bridge being built to the highest standards we know how and someone should be liable if it falls down. You can rely on this in real life. Why do we not have this for software?
I'm not saying I write all of my software using formal methods and rigorous proofs. It's often good enough to sketch my idea on the back of a napkin when I'm building a tool shed. But nobody built a sky-scraper without doing a little math and being fairly rigorous in their thinking.
You're disagreeing with my 'I can't imagine'? So you're saying you've built non-toy systems with TDD where you've never tested anything but the user-facing behavior of the system?
I'd be interested in knowing how, and what kind of thing you build. I can almost imagine it for an API or framework, where almost every implementation choice is on the boundary. But even then, you have to test private functions to do TDD, surely.
I work on real time stuff, and in that stuff, the user-facing functionality is hard to test, and usually not tested in a TDD kind of way. The TDD stuff is all internal.
>I've seen situations where that big refactoring would involve scrapping or rewriting tests, and that causes its own kind of architectural conservatism.
The way to avoid it is to either be damn sure that the API you've surrounded is tight, clean and unlikely to need to change much or to test at a higher level.
I understand "implementation decisions" in quite well-defined way; only functional invariants should be tested. Your example is a perfect illustration for one.
Additional point is that invariant-based testing can easily be transformed into automated fuzzy tests using Quickcheck and its analogues, and this can find immense amount of otherwise hidden bugs.
"I've seen situations where that big refactoring would involve scrapping or rewriting tests, and that causes its own kind of architectural conservatism."
Yes, rewriting tests is part of a refactor. You always have to be willing to do that or just scrapping outdated tests.
I come to this from a test/automation engineering background and I'd like to amplify your last point.
There are a couple things to consider that sometimes become taboo in orgs--but shouldn't.
The first is that you don't need to test everything. I recommend having a great set of those API-level black box tests as an acceptance suite, but at a component or unit level you should pick your battles. If you'd architect for DI -only- to test, that testing had better be pretty necessary, otherwise you introduced complication (and risk) without a lot of benefit. That risk can actually overwhelm any benefit of testing.
The second is that you should absolutely scrap tests of implementation details at the point that the implementation becomes a black box. Tests are like scaffolding during construction, and we don't leave the scaffolding up when the building is done--we do inspections of the building instead. The same mindset should be applied to testing. Different phases need different tests, and too many tests at too low a level of abstraction absolutely cause maintenance and flexibility issues if you're not willing to let them go.
But that's no reason to back off entirely. SOME things need to be unit- or component-tested SOMETIMES. Those things generally benefit from TDD.
It's all about making intelligent decisions. There's a sometimes-controversial movement in QA/test called "context-driven testing" which comes down to there are very few always-best practices, and strategy should be tailored to your problem and situation--not done by rote, book, or prior (read, your boss's or CTO's) assumptions of what's always right. Otherwise you tend to get something more onerous than useful.
The considerations around that approach apply at the low-level test tier as well.
From what I understand author is not questioning the usefulness of testing, just TDD approach of writing tests first. I personally prefer a sandwich approach, I start with code first, and then write tests for it as soon as I have that little piece of logic finished (usually a method or two). Then I add more code, followed by more test, and so on. Works great for me and my team.
I nowadays do an approach where I don't write tests first, but I write down the test titles (I use a language where the testing framework gives a nice DSL to write nicely readable test names compared to method names) and I just mark them as pending (which will report as no test failures or passes but ignored cases). I rarely end up with the same test ideas and names which I started from, but since I only start with names which do not hinder refactorings this worked out surprisingly nicely for me.
I agree. If people started viewing coding as an art instead of a strict science I think people would feel more comfortable doing something like this instead of a strict Test First or Test After.
My impression, too, the author explicitly says he understands value of unit testing and will continue to do it, but he's going to give up on the idea of writing tests before he writes the actual program code. Strange, from what I can tell 90+ percent of comments in this thread treat the author as if he said he was giving up tests entirely.
One good way to the limit breakage in such cases is to solely perform black box tests on the API level. In case of our Node.js based Backends we don't ever write a single classical unit test, instead we have a custom Framework built on top of Mocha which performs tests on the HTTP layer against all of our endpoints.
This works remarkable well in practice and allows for large scale refactorings under the hood with little to no impact on the tests. We can also mock databases, memcached, redis and graylog on their respective http/tcp/udp level. This in turn means no custom build mocks which could break when refactoring. The tests itself also contain no logic, they are pretty much just chained method calls with data that should go in and an expected response that should come out, along with a specification of all external resource our API fetches during the request and their responses etc. Any unexpected outgoing HTTP request from our server will actually result in a test failure.
As for scaling this approach, from our experience it works quite well, especially when you have lots of complicated interactions with customer APIs during your requests since the flows are super quick to set up.
Yeah, I mostly prefer end-to-end tests as well. Though to be fair, they are often slower than unit tests, because you need to start up the whole system. And they are worse at pinpointing problems, though that doesn't seem to be a big deal in practice.
I in part agree with the pinpointing, though in my experience this really boils down to an issue of scope and how much of the data you want to test in each of your tests. E.g. an API returning user data, do you have one test for the whole set of data or one test per field.
For our use case we have some pretty "fancy" deep-equals logic for nested structures and allow to specific fields as "to be ignored" in our test expectations.
When it comes to speed, the most important part is to cut out everything you don't need, for us this means that our testing framework completely throws away the internals of Node.js HTTP layer. We never create a single TCP socket in our tests, which gives an incredible speed up. As a bonus this also allows to test timeouts and low level HTTP errors in < 1ms, since we can just return the timeout directly from the low level APIs and Node.js will invoke the timeout event on the http client handle.
And yes, that really reads 10 seconds and yes I always get a bit bored when I have to run the tests for one of our Django backends, feels like an eternity... :)
Count yourself lucky that your test takes 10 seconds. A partial build on my project takes ~3 minutes to compile + link. A full build is about 3 hours for everything.
I recently cut some link times from 3 minutes to 14 seconds by switching linkers (from bfd to gold doing Android dev.)
...sadly that's still only for a single project x config x arch combination. I still need to play around with incremental linking options that appear off by default...
I had a build that took 25 hours to run once. Never again. The only reason we got any work done is that we ran a 1, 3 and 19 hour part of it in parallel. But I really just wanted to throw out the long part (bad E2E tests) and start over.
I like to test the whole system via end-to-end tests, as they're the best bang for the buck. And then I'll create unit tests for more algorithmic code, like a parser, sort algorithm, shortest path calculator, financial calculations. Those also tend to require the least amount of test context setup, making them less painful to write.
I have almost always worked for customers who enjoyed changing their minds arbitrarily and often in ways they swore they would never do.
In the face of grossly changing requirements, I've never had much luck keeping E2E tests up and functioning properly. And people have a bad habit of investing more time and energy than a particular test is worth in trying to keep it working or porting it to the new requirements.
Unit tests are cheap. If the requirements change invalidates twenty of them, you just delete them and write new ones. Easy.
I took this approach on several different projects, but I figured that a lot of the boilerplate/infrastructural code that you need to actually write these kinds of tests is poor or simply not available.
For example, declaratively starting and running multiple services together, in parallel and at the right time (with service dependencies) and printing their logs out.
Or, mocking the forward passage of time. (click something -> move forward a week -> click something else).
Or, asynchronously 'listening' using epoll for emails on a mock SMTP server that logs out the emails it receives.
Selenium's good for the web interaction stuff, but you need much much more than that to be able to effectively test at this level.
I think this kind of testing would be much more widely used and effective if the tools available were up to scratch.
I'd say do both. Unit tests often help me to make my code more readable/decoupled and also help to spot potential problems early on. They also act as a kind of abstract documentation for how things are supposed to behave/work.
But functional/integration tests are what matters most in terms of being confident of deploying big changes because you can ensure that all the endpoints that are actually consumed by clients work as they should, but it doesn't help too much with spotting problems on the code level.
For smaller projects I definitely agree with having both unit and integration tests, especially for libraries. One thing to look out for is the fact that you can always "cheat" in unit tests, e.g. you can be "lazy" and set up some internal state directly in the test to skip huge amounts of initialization, this of course becomes a problem when there's no actual integration test for making sure that the exact state can also be triggered from the outside when using the API. In my experience, ensuring that these cases are always covered can become pretty complicated once the project grows.
In our special case we have about 100 different endpoints all versioned and all dependent on multiple endpoints from the (rather badly documented) APIs of our customer. Most of the work our API does is spent combining / enriching the customers data and performing integration across the subresources. Setting up individual mocks for every single on of these complex requests flows manually is pretty much impossible at this scale.
So doing black box testing and enforcing a 100% test coverage (best for avoiding dead code) helps keeping us sane. In the end we don't care so much about how the implementation behind our HTTP response looks as long as we return the correct data in the end. The code itself still has to look good though :)
yeah, well you just test what you own anyway, for other stuff you could just grab some kind of dummy response and use that for testing your manipulation of that data, but of course you have to trust your customers endpoints to return the data in the correct format because that is out of your control.
We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.
These have become known in Pivotal as "frenemy tests". Poke a remote API to see that a) it is running b) it hasn't dropped a world-stopping change.
Typically run as a pre-build sanity check when you have remote integrations.
On some Labs projects we've written extensive request tests and stub services based on documentation, then handed those to the upstream service. Usually there is some angst at this point, being the first time that any kind of TDD suite has turned up to ask awkward questions.
I still maintain F5 owes me a job. I did QA for them for a year back in the dot-com boom. The first couple of versions of BigIP didn't really support session affinity, despite being the flagship feature. I think we filed something like 6-8 pretty big bugs, all in different parts of the code.
Sadly, despite not working, they were still about 4 years ahead of their competitors.
In my experience, for any non-trivial application with a non-trivial number of tests, some percentage of those tests, particular the unit tests, will have by accident or carelessness come to rely on internal implementation details that aren't reflected in the result.
Those kinds of "atrophied" tests can make a refactor considerably more painful than it should be.
This is exactly why we created Newman (https://github.com/postmanlabs/newman/). Hitting every endpoint of our API gives enormous confidence when deploying it to production.
(Disclaimer: I work on Newman as a part of my day job)
Functional tests are the best kind of tests. While unit tests are nice sanity checks when implementing tricky methods, when the rubber hits the road, you want to know if the whole app works as you expect it to.
If one test breaks you get a nice message. If 20% of your tests break because they depend on functionality that you just changed intentionally, you increased the cost of your feature a lot.
If you changed that functionality intentionally, then the other things it interacts with will require a change regardless. Furthermore, if it's that small of a change(as usually, the smallest changes are the ones that impact the most tests), it usually doesn't require any change to the test, aside from adding/subtracting a parameter or something.
Major functionality changes should carry weight. It's like saying that a motorcycle runs so much faster without a person on it- it defeats the purpose.
I suspect the author of the article is only giving up on test first or test driven development where you write tests before writing each bit of the program. I doubt he is against having a corpus of tests for your program.
That's different, I'm against TDD for the same reason the author is. I'm FOR automated testing for exactly the same reasons you are.
Think of it this way: pre-testing makes you afraid to change you mind because you throw out your tests, not post-testing makes you afraid to change your mind because you might break something. If you think changing your mind is good the path forward is clear.
I think the language and in particular the type system matters a lot. So making big changes in the absence of tests in a Swift or even the somewhat more loosely typed Objective C is less scary than in Ruby and Javascript application.
Agreed. I'd argue that if a large number of tests break when a "large" change occurs in an application, those tests are either testing the wrong thing, or written with bad starting assumptions. Yes, there's a transitional period where the responsibility may shift from the caller to the call-ee to take care of some resource, and yes, stuff will break, but the amount of test breakage should be proportional to the impact to the application.
I greatly-prefer the warm fuzziness of knowing that I have a large number of unit tests as a "safety net" to detect if my own thoughts on what is "correct" are in some way wrong.
> Under these conditions, large scale refactorings or framework upgrades can cause massive test failures
An even bigger problem is that large scale refactorings can result in the tests themselves no longer being correct (which isn't the same thing as “not passing”), if you test at too fine a level of granularity. However, when you wrote these tests, you couldn't have possibly foreseen a large scale refactoring 18 months into the future, so how do you tell in advance what the right level of granularity for your tests is?
When tests function as a type system as well, this is true. Buy a good type system will guide refactoring and a good module system and IDE and will guaretee isolation .
Ruby and Javascript are dynamic languages, so you need all those tests to make up for missing compiler type checks. With staticaly typed languages like Java or Scala things are much different.
In my observation and feeling, the statement:
" Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests."
is most likely due to the low quality and low test coverage of the exiting code base. So that the tests are fragile, seemingly hinders changes.
If things are started with TDD, I would be really surprised that tests hinders change.
While I understand that the conservatism the author refers to might be detrimental to the individual working alone, in a team environment it is actually an advantage. The module the team member is working on is likely a dependency for some other team, and any test-breaking refactoring---however beneficial it might seem---needs to be conscientiously approached and coordinated with other people. There may be very good reasons not to pursue refactoring at the moment, reasons that an individual narrowly focused on the module at hand might not be aware of.
Don't expect just because there are no test failures that you haven't broken anything. I can't stand developers who have this mindset.
Just because there are unit tests, doesn't mean they are covering all the code foremost, or that they are written properly to truly detect when things have changed for the worst.
Tests are not a replacement for actually testing and verifying your changes in whatever application gradient you're working in.
I too don't have the disposition to patiently wade through fixing tons of tons of broken tests over and over again. IF I have to I can do it, but I know better than to unreservedly trust my judgement about how things are progressing.
But what this has taught me is to stop trying to push water uphill. If testing is hard because of questionably coupled code, refactor it NOW instead of waiting for things to get bad. The refactoring almost always suggests new features we could add to the code (or makes me backpedal on pronouncements that certain things were 'impossible'), and the number and kind of tests that break is reduced.
I came across a quote recently from Bertrand Meyer (the Design By Contract guy), where he suggested that code for making decisions and code for acting on those decisions should be separated. I found myself nodding along to this because it was something I knew intuitively but had never articulated: Decisions without actions don't need mocks, and actions without decisions need at most one or two (often none). Decisions can be tested (possibly in parallel) with unit tests, and actions tested with functional tests. Then all your integration testing is that decisions lead to actions (basically that event dispatch works) and that catastrophic failures are handled gracefully. A proper testing pyramid.
Now I want to figure out which of his books or interviews this was in because I want to read the rest of what he had to say on the subject.
Like all things in life TDD should be taken in moderation.
It's an excellent process to create stable and maintainable code, but it does not fit every bill.
But abandoning it completely on the grounds that it sometimes makes you write "bad software" is a bit weird to me, in fact, one of the main arguments for TDD is that it makes you write better code.
I found that it does make you write better code many times. So like all things, use when appropriate.
I guess the real hard thing is to determine when using TDD is appropriate.
I like to think that, like with any other technique, with experience comes the ability to decide when not to apply it. Any old tutorial will show you an example of when it works, but only with experience will you learn when it might not.
> Like all things in life TDD should be taken in moderation.
You're points are all good, but your opening one especially.
The obsession that some developers have with methodological purity can be really puzzling sometimes.
Where I'm at: I work in research and engineering (in the gov't, on the borderline between industry and academia), and I can say that doing any sort of testing at all (either up front or later on) is an improvement over the untested, undocumented, unversion-controlled status quo that exists in a lot of cases.
I don't understand why people tend to turn useful things into strict ideologies.
TDD is super useful for simple algorithms. It gets harder once you get into more complex scenarios like multiple components working together. If TDD is not useful for some cases then don't use it there and use it where applicable. Or think about how you can make it work. It's not that hard.
I think the first and fourth points are applicable to unit tests in general rather than just TDD.
For the first point, I'm reluctant to make large-scale changes to code which doesn't have lots of unit tests because the unit tests give me confidence in what the behaviour of the system should be.
Certainly the third point is something I try to bear in mind when doing TDD. I've found that having someone do a code review after a feature is complete gives someone the opportunity to come in from that high level and look at the program as a whole and check that your design makes sense.
TDD is nuts for code without a client or specification. The whole point of tests is to ensure that the code does what it's supposed to do. When you have neither client nor spec, how are you supposed to know what the code is supposed to do? There is, IME, a >90% chance that any such code will be ripped out and replaced as you develop a better understanding of the problem domain.
I've found it's pretty useful to go back and add tests as you accumulate users, though (or convince an exec that your project is Too Big To Fail in the corporate world). You're capturing your accumulated knowledge of the problem domain in executable form, so that when you try an alternate solution, you won't suddenly find out - the hard way - that it doesn't consider some case you fixed a couple years ago.
I don't agree with this at all as the tests at least say what the code does in its current state. I see that as incredibly useful. It also forces you to think about what it's actually doing as you build it. I get the OP's forest-for-the-trees thing, but I still find incredible value in testing things - even my side projects.
Imagine you make some code that needs to comply to a law, but then the law changes. Or client decides he wants his notification in different format, look shape.
And you've spent 2x the effort that you would've had you not written the test in the first place.
My initial comment was meant to be a statement about the relative chance that you're solving the wrong problem vs. solving the problem wrong. Prototypes help you identify that you're solving the wrong problem, unit tests help you identify if you're solving the problem wrong. In the beginning of a project, you are much, much more likely to be solving the wrong problem than solving the problem wrong. Go ensure that the system works end-to-end and solves the user's needs before you make it bulletproof.
Some (quite a lot, actually) pieces of code are essentially experimental - e.g. trying out an API/libary and seeing what it's capable of or trying an approach to solving a particular kind of problem or even trying to see if a particular problem is solvable with a piece of code.
For this kind of coding, TDD makes no sense whatsoever. The 'specs' are as fluid as the code and having confidence in the code isn't that important.
This is entirely different to creating production hardened systems with very clear specs. If you don't do TDD on that, you're an idiot.
I don't agree with "no sense whatsoever." Actually, TDD can be a very pleasant way to do this kind of exploratory programming, precisely because it's oriented around verifying expectations.
Yep, any time I have an assumption about how code should work, that's a good starting point to write a test. Even if it's vaguey like "should not throw when given inputs X Y Z that I expect will be encountered"
That's assuming it works correctly the first time (I haven't had that experience often, even for "trivial" code :( ). Even for "run once" functions, I still use a few tests to develop them and make sure my expectations are correct. With a good framework, setting up a handful of unit tests takes just about as much dev time as running the function in a REPL.
I'm not clear on what the scope of selenium is or how it works, but for a general web scraper, I'd identify some targets I want out of skyscanner. Here's a quick googletest butchery for "I want to make sure my function returns flights for a known good flight search."
The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings. People "know it when they see it" without necessarily being able to define it.
So you write the code; expose it to a preference function / oracle, iterate, and hill-climb towards a local optimum. The revealed preferences may give enough information to make a creative leap off the local hill onto a large hill elsewhere - but it may not.
After a solution has been found, then you can write tests.
> The preference function (or oracle) can return a truth value without you necessarily having insight into its inner workings.
Is this preference function implemented on a computer, or is it just a person giving you seemingly random answers in response to whatever you tell them?
> People "know it when they see it" without necessarily being able to define it.
That path leads to hell.
> After a solution has been found, then you can write tests.
what exactly would you test? Tests make sense when you want to check that your implementation conforms to a certain set of logical rules, but where exactly is the logic in the process you outlined?
> > People "know it when they see it" without necessarily being able to define it.
> That path leads to hell.
That's why they pay us the big bucks.
If people could define their problems well enough without iterative design, waterfall development would be a success story and all software would be outsourced to the lowest bidder.
The requirement for an iterative process is what I mean by "oracle"; it's the thing that tells you how far off you are on this iteration, and how to change your heading for the next iteration.
That's a rather weird use of the term “oracle”, which normally means “a black box that solves a decision problem in a single step”. Normally, to decide whether you've achieved your goals, you don't use a black box - you use performance metrics that you've defined yourself (and thus can't claim not to understand).
In XP settings this is called "spiking" and is recognised as an entirely legitimate tactic.
The point of spiking is that you don't know enough to TDD. You are outside the sweetspot where TDD is tractable.
The key to spiking is that once you come with a plausible approach, or have a conceptual breakthrough, you stop, backtrack and then switch back to TDD.
Like everything else in mature software engineering, agile or classical, it requires sustained discipline.
Interesting. I've found myself drawn into XP-style delivery repeatedly, probably because I tend to work closely with clients who have rapidly shifting or unknown requirements.
I've found the opposite, that TDD helps me most when the spec or my understanding of the spec is fuzzy, by forcing me to make decisions about behaviour up-front and clarify my thinking. Otherwise I can get bogged down trying to implement and specify a feature simultaneously, or spend a lot of time implementing a feature before realising I'm approaching it the wrong way.
Depending on how you work, that can be worthwhile, but I usually find I get more useful information quickly by making the decisions that make implementation easiest, getting something up on the screen that I can react to as a user (or put in front of a real user), and then seeing where that initial proof of concept falls short. In general, I've found that pushing off decisions until I have as much information as I can tends to result in better decisions.
how are you supposed to know what the code is supposed to do?
That's the beauty of TDD. It forces you to sit and work out, "What is this code supposed to do from the client perspective?"
It battles the all-too-prevalent developer inclination to just start running with implementation without even taking the time to understand what problem is being solved.
> TDD is nuts for code without a client or specification.
So, write/pretend to be one? I find writing high-level overview docs, and API interface example code before writing code gives enough guidance for my personal project to then write some tests & then code (in any order).
One of the things that people aim for in writing tests is orthogonality - different tests should not break for the same reason. This promotes the ability to refactor and change your code. I have also seen massive codebases, with masses of tests which were rarely run, and which effectively concreted the code and stopped it from changing.
But doesn't test redundancy reduce the risk of a test isn't testing the thing you thought it was?
If the same situation is tested in 2 different ways, a bug in one test might cause the test to fail to correctly handle all case, but the second test might still catch.
Maybe auto-generation of test cases would be a better technology?
Typically tests can provide fault detection and fault isolation, but to different degrees.
A feature/e2e test typically provides the most effective kind of fault detection, because it combines all the components of the real system with no stubs or mocks. But if it shows a fault, then typically that fault is hard to identify, because it wasn't previously driven out by a more detailed integration or unit test.
Contrariwise, integration or unit tests will typically be best suited to isolating faults, but typically those faults are limited to the class of things you included in your tests.
This is why we have "test pyramids": a handful of slow, brittle feature tests at the top, then an increasing volume of faster, less-brittle integration and unit tests.
TDD is almost orthogonal to the testing pyramid, with one difference from non-TDD tests. In TDD each line of code was, ideally, driven out by a test or tests. Fault detection is increased, because each line was written in response to a manufactured (test-first) fault.
Over the years I've gone from writing no tests at all, to being a die hard TDD purist, and then out the other side to writing some things with tests first, others with tests after writing the code, and some without any tests at all.
In some situations I have clear view of what I need to build, and how that should work. TDD is great in that case - write a test for the expected behaviour, make it pass, refactor, rinse and repeat. The element I think a lot of people miss when doing this is higher level integration tests that ensure everything works together, because that's the hard bit, but its also essential.
Other situations you're still feeling out the problem space and don't necessarily know exactly what the solution is. There's an argument that in those cases you should find a solution with some exploratory development then throw it all out and do it with TDD. If I've got the time that's probably true, it'll result in better code simply through designing it the second time with the insight provided by the first pass, but often that just isn't viable. Deadlines loom, there's a bug that needs fixing elsewhere, and I've got two hours until the end of the week and a meal date with my wife.
Finally there's the times when tests just aren't needed, or they don't offer a decent return on the effort that will be required to make them work. I'm thinking GUIs, integration testing HTTP requests to other services, and intentionally non-deterministic code. Those cases certainly can be tested, but it often results in a much more abstract design than would otherwise be called for, and brittle tests. Brittle tests mean that eventually you stop paying attention to the test suite failing because its probably just a GUI test again, and that eventually leads to nasty bugs making it into production.
One thing I'll directly say on the article is that I found his opinion that its hard to write tests for handling bad data. That's almost the easiest thing to test, especially if you're finding bugs due to bad data in the real world - you take the bad data that caused the bug, reduce it down to a test case, then make the test pass. That process has been a huge boon in developing a data ingestion system for an e-commerce platform to import data from partner's websites as its simply a case of dumping the HTTP response to a file and then writing a test against it rather than having to hit the partner's website constantly.
Over the years I've gone from writing no tests at all, to being a die hard TDD purist, and then out the other side to writing some things with tests first, others with tests after writing the code, and some without any tests at all.
Hear, hear! Exactly the same here, for the same reasons as you and the OP mention.
And observed fom a distance, it's always the same universal principle: purist behaviour (in the sense of almost religous beliefs that something is a strict rule, sentence starting with Always etc you get the point) in programing or to a further extent, in life, is nearly always wrong, period. No matter what the rule is, you can pretty much always find yourself in a situation where the rule is not the best choice.
I deeply agree when it comes to programming, but there's plenty of room for absolutism in other areas of life. Why subordinate strong moral preferences to hazy cost-benefit analysis?
This is becoming very off-topic, but if absolutism doesn't apply in the highly-ordered world of programming, to me it's even less likely to apply in messy real-life.
> you take the bad data that caused the bug, reduce it down to a test case
That's not really TDD anymore though - post-hoc testing of bad data is always going to be orders of magnitude easier because you have the bad data and you know what broke.
> ... it encourages you to make design decisions that can be tested rather than the right decisions for the program users...
This is exactly why I don't do TDD. I prefer to think about the design and implement features in non-distracting way. After I have it shaped, I add tests to avoid regressions.
Where I do write tests first is when I need to reproduce a bug, so I have the test, again, to prevent regressions.
Just like the chicken and egg, it doesnt matter which comes first, code or test. The key is that both are written, ideally around the same time and part of same changeset. Refactoring posthoc for testability is tricky and often brings to the surface poor software designs in the original implementation - bad coupling, module dependencies, leaky abstractions, etc.
I think there's something to be said for maybe saying test first. I tend to find writing tests second a little harder. If its a big complicated feature, its much harder to go back and try to think of all the test scenarios needed when its finished, there might be an edge case or a semi obscure case that may have been apparent at writing the code but gets missed when looking back and writing tests. But its preference.
I recall a situation where a manager was pushing for TFD in an Agile environment. What a nightmare, the two just don't mix. The biggest issue isn't the approach itself, but when it hits Buzzword level with the managers who have no idea how development really works.
This is true. These kinds of changes should come from the professionals -- it is, after all, their reputation on the line. As a software developer and executive simultaneously, I walk a fine line -- usually I adopt a practice myself and show people what it has done for me and the maintainability of my code. Then if people are curious I help. If not, that's fine. They'll come around or go somewhere else where engineering is less rigorous.
It's extremely tricky to avoid being threatening as a leader. Your position gives you power nobody else believes you deserve. Even when, sometimes, you do. Not saying I deserve it, but there are those that I've worked with that did, and I only noticed in hindsight. F'ing limbic system.
In my scenario, not only was the management mandating TDD, they were also mandating a code coverage minimum, leading to joyless soul-crushing unit tests such as those for getters and setters to boost coverage. The projects ultimately were not successful and this same swarm is I suspect now making some other developers' lives very miserable with the same broken-record absolutist ideologies.
It's a way of offloading more work onto the coders so that the PMs can just go down a checklist instead of having to actually manage. Now instead of just constantly refactoring, you need to write new tests each time first. Then it's sold to us as making us "better coders." It doesn't make us better coders. It makes us easier to manage and puts even more work on our plates.
The author keeps talking about a "better design" achieved when not using TDD, and he goes on saying that sometimes a good design is a design that is hard to test. These are subjective, psychological feelings that, in my experience, bring only to maintenance nightmares and impossibility of doing any kind of refactor without the fear of breaking something. TDD might be difficult to do at times, but it has a very interesting, objective advantage: it brings the "good design" buzzword out of psychological and subjective interpretations. With TDD, good design is testable design. Period. Is it easy to check? Yes. It is objective? Yes.
Testable design is often a poor design, mainly because of language limitations; abstraction and scoping for tests are not necessarily the best choice for abstraction and scoping for design.
public class Foo {
private static class Bar {
public void someMethod() {
}
}
}
Unit test someMethod.
More realistic, concrete examples are harder to describe because they involve an interaction over time between a somewhat vague problem description and exploration of a solution state space, where the abstractions chosen are fluid and slide around before they get into a good shape.
I, for one, tend to write code from the bottom up, i.e. creating hypothetical abstractions, small tools etc. and start composing them to solve the next problem up the abstraction stack. Then, when I find it doesn't fit quite right, I adjust, freely throwing away abstractions, rewriting them, reshaping, until the level of my tools fits the problem better. I gradually build up my abstraction level until I can move problem domain level mountains with little effort. Doing this in the form of tests just doesn't work (for me).
Writing code from the top down isn't the right answer either; not all people think that way, and besides, it can lead to solutions with very ugly implementations - the grain of the wood should influence the design of the house, if you will.
As Uncle Bob describes in his book, you never even make abstractions until you have proven duplication. You make your abstractions based off of evidence, not hypothetically planned out.
Kent Beck in his book doesn't recommend top-down, or bottom-up, but rather from known-to-unknown. Start with what you know, and work toward what you don't know.
Edit: The books I refer to are Test Driven Development: By Example - Kent Beck, and Clean Code - Robert Martin
Abstractions aren't merely for duplication removal. Abstractions are for symbolic chunking; for thinking about things at different levels.
We don't use e.g. units-of-measure types [1] because they remove duplication from our code; if anything, they add duplication. They do however clarify our thinking with help from the type checker.
I think a lot in terms of flows / pipes / functional transforms. So I tend to try and express problems in that shape, because I have a lot of mental tools that I can apply, and I know they're extremely easy to test in isolation. Creating a pipe-like thing means reducing it to a simple common push or pull stream pattern. But it's not duplication I'm removing here; I'm actively introducing an abstraction because it has proven power for creating good software and solving problems.
I try and create few, minimal abstractions that can be applied widely. Take a cue from functional languages and split out types from algorithms; if you make your types more general, you increase the reusability of your algorithms. Classical OO design tends to create a lot of types that are specific to the domain model. I happen to think that OO designs are usually poor; they tend to have a high code complexity to implemented logic ratio, and require awkward composition that leaves details hanging out.
I spent 20 years writing OO software and was a big fan, especially in the early 2000s. I still work in OO languages, but most OO code I read makes me wretch now.
I don't recognize either Kent Beck or Bob Martin as particularly noteworthy for good architectural design (Beck is on the right path with agile though). In fact, I blame TDD for Java-itis: proliferation of single-implementation interfaces (has there ever been a worse idea more often propagated by cargo culters?), poorly abstracted object graphs, leaky implementation abstractions, and more.
> Start with what you know, and work toward what you don't know
I've been programming for more than 25 years. A lot of stuff has changed in that time; what hasn't changed is patterns of abstraction. So I start out with abstractions, chosen from experience.
Read Norvig on TDD [2] - his experience matches mine. If you know a lot of software tools (i.e. abstractions, algorithms, approaches), you can apply them to a problem. TDD is a poor tool for creating new tools, though. And if you rely on code duplication for creating a tool, well, you're going to have a bad time with hard problems.
> Sometimes, the best design is one that’s hard to test
I strongly disagree with this statement. The best design for your program is always the one that's easiest to test; the one that's modular, the one that separates out dependencies, the one where methods are as atomic as possible.
If you find yourself wanting to write code that is hard to test, then you are approaching the problem (and/or the solution) the wrong way.
> The ‘purist’ approach here, of course, is that you design data validation checks so that you never have to process bad data. But the reality is that it’s often hard to specify what ‘correct data’ means
Again I disagree. It should be really clear what correct data means at a low level. If you can't, then you haven't fully fleshed out your design, so yeah it's gonna be impossible for you to test it.
I'm far from a TDD zealot. But on my team we adopted writing unit tests for everything in the last 6 months and it has been night and day. It is incredibly useful and the resulting code is so much better.
I think a lot of the author's problems stem from attempting to do this alone and not having someone else to guide him through how to tackle things that he's having trouble with. Plus, frankly, he seems to just have a defeatist attitude about the whole thing.
TDD only works if the tests are written correctly.
Tests are not about "code coverage", nor about establishing the exact sequence of things in stone. Tests are about fixing invariants.
When a new project starts, I only know about 10-15% things for sure, and those are exactly which will go in tests, before writing any new code. I don't worry about some things in my code are not yet covered by tests; I don't know yet how they will turn out, so I can't write any meaningful invariants.
In my experience, useful tests are much higher-level than TDD guys prefer. They routinely fix invariants for the entire system / subproject, not assuring coverage of every method in class (some crazy folks are even testing getters and setters — why?)
> some crazy folks are even testing getters and setters — why?
Well, I can see the logic if your getters and setters are hiding more activity than simply retrieving/setting the value of a private field, which is the point of having separate getters/setters at all.
If you imagine a getFullName() / setFullName(name) pair, for example, that actually reads from/writes to two different private fields for first and last name (leaving aside middle names, internationalisation, etc), then there's some minimal logic there that you might want to test.
In a duck-typed language, when you're trying to ensure a class obeys an implicit interface, it may also have value.
Apart from that, for vanilla getters/setters, it's a little pointless.
In such case, a correct test should focus on functional invariants, like getFullName() + " " + getLastName() equals to getName() (btw, never do that — names are much more complicated than that. In some countries, there are no first and last names at all; others use multiple name designations; some have meaningful patronyms etc.)
I agree (and implied) it was not a great design for actually dealing with names, but simply that sometimes there is more in getFoo() than "return foo". Which behaviour you may want to verify with a test.
* fixing bugs - reproducing the bug in a test case both confirms you're fixing the thing, and acts as a regression test
* defining a protocol - where you need to glue two things together, e.g. a front end UI and a back end controller, or a model shared between different modules
It's a lot weaker for design. Test-first tends to encourage overly open to extension abstractions, because you need to make things visible to tests and make components replaceable by mocks. In the early stages, the weight of updating tests makes the design overly awkward to change. And early on in the process is exactly the wrong time to be creating your general abstractions - that's when you have the least amount of information about what will make for the best abstraction.
You still need to back-fill tests after good abstractions have been chosen, of course. Tests are great; test-first, specifically, isn't always best.
As a suggestion, QuickCheck type testing frameworks are good for finding bugs relating to unexpected data.
Summary of how they work: you say "this program takes a 32 bit signed int and a string" and the testing framework will throw it a heap of random ints and strings, some of which match the sorts of classic curve balls it might encounter (negative numbers, int max and min values, strings that are very long, empty strings, strings with \0 in them, strings that don't parse as valid unicode, "Robert');DROP TABLE Students; --", and so on.)
In my limited experience, tests should be written against a specification, not against code. Writing tests focused on code seems to often lead to tests that are too close to the implementation of the program, even with test-first approaches (you end up thinking about how you'll write the code when you're designing the tests).
By focusing on the specification, you can ensure that design decisions are made by the coder are fit for purpose.
I also think many tests would be better written as code contracts, you mainly want to ensure the inputs and outputs are valid, code contracts focus on this.
> I won’t spend ridiculous amounts of time writing tests when I can read the code and clearly understand what it does.
Is he arguing that simply trusting yourself not to make mistakes is a sufficient guarantor of quality? Ian Sommerville is the author of a famous textbook on software engineering, so it would be surprising if he was.
TDD is actually much more difficult in practice than people realise. I read Kent Beck's book and thought it sounded like utter horseshit. I tried doing it myself and decided it was definitely horseshit.
Then I came to work at Pivotal Labs. Now I am distrustful of code that was written before tests.
As for the argument that TDD distracts from the big picture, this is like saying that indicator signals and mirror checks distract from driving. Sure, when you are learning to drive, you feel so overwhelmed by attending to all the little details that you struggle to drive at all. You become unable to focus on your navigation.
After a while you learn the skill and it becomes automatic. TDD is such a skill.
The bread and butter of Labs is teaching people pair programming and TDD. So I was taught those things, and I'm still learning.
It takes a few passes through the whole cycle of story->feature test->integration test->unit test for things to begin to click, for that way of thinking to become habitual. It is uncomfortable and frustrating for some time, as Ian Sommerville found. In fact, because most code is not written test-first, it is difficult to test. So all that keeps you to TDD is habit and, to some degree, help from another engineer.
> And what do you get with regards to specs? It seems to me the methodology fits how requirements are derived.
The most popular story-writing style I have seen so far is the As A/I Want/So I; Given/When/Then. Usually some acceptance criteria are supplied. To varying degrees of fidelity that the feature tests will typically follow Given/When/Then.
In our weekly Iteration Planning Meetings engineers are expected to pipe up if there are useful ways to decompose stories into smaller stories, which makes the overall situation more manageable.
When writing tests is boring, difficult, and tedious, that's a really good time to think hard about the way the program is structured, if you have time for this.
The way to make testing pleasant is to extract more and more behavior into units with clear boundaries... realizing how to do this was a major event in my programming career, and I attribute the insight partly to doing TDD.
I don't agree with some posters who say that TDD encourages overly generalized design. Sure, it encourages some form of dependency injection... but mostly, it just encourages the creation of coherent and loosely coupled units, which is a universally lauded best practice.
When I finally decided to really decouple and mock dependencies and create "pure" unit tests was the major "Ah ha!" moment of TDD. I had been trying it out here and there but never really saw the benefit because I wasn't really creating nice testable units.
It is hard to say for sure, because the author doesn't give any specifics but wording like:
"You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests."
make me think the author wasn't actually making unit tests, instead they were likely end-to-end tests, or partial ETE tests that were running inside a unit test framework. I have had many disagreements with other developers that just because your test runs inside a unit testing framework doesn't actually make your test a "unit" test.
Sure, and that fixing is the process of making sure that the code still works, and that you still have the test suite as a second client of the code.
As with everything there are costs and benefits!
A lot also depends on the quality of your tests. If the team doesn't care about test quality or doesn't know how to write good tests, you can easily get these big blobs of incomprehensible and tedious tests...
It encourages dependency injection, which in my experience will eventually encourage a functional programming style since there's so little barrier to it once you're doing things like injecting the current time into a method which makes use of it. Maybe I'll turn around in five year's time and regret saying this, but I've never regretted pushing a project in a functional direction.
Most people probably felt the same way after only a "few months" (best-case, perhaps less) of practicing TDD.
And certainly TDD is harder as you approach the GUI, you want to test in vague ways which don't break with every change. If you thoroughly test all of the underlying behavior, implementing a GUI is typically incredibly trivial because everything beneath it is known (and proven) to work. Most of the article is not related to the GUI.
> ...it doesn’t work very well for a major class of program problems – those caused by unexpected data.
This is a hollow argument. Regardless of development methodology, if unexpected data isn't considered at all it could have all kinds of side effects.
Regarding conservatism with breaking tests (many tests failing for one change), it's likely the result of a structural problem within the application if it's an intimidating number of failures.
> It is easier to test some program designs than others. Sometimes, the best design is one that’s hard to test so you are more reluctant to take this approach because you know that you’ll spend a lot more time designing and writing tests (which I, for one, quite a boring thing to do)
Not sure how this applies to TDD, if you're writing tests first you aren't deeply concerned with designing tests because you're imagining what the interface for well-designed code would be, and then you write it. It frequently sounds like the author jumps into writing tests without any forethought.
> In my experience, lots of program failures arise because the data being processed is not what’s expected by the programmer. It’s really hard to write ‘bad data’ tests that accurately reflect the real bad data you will have to process because you have to be a domain expert to understand the data.
If you don't understand the variety of inputs, how can you possibly validate them? Programmers should have some domain understanding, certainly program inputs fall within that realm.
> Think-first rather than test-first is the way to go.
I agree; but step 2 should be testing in my opinion. Test first is just the first tangible work product, it isn't a ban on thinking.
I deliberately decided to experiment with test-first development a few months ago.
and
the programs I am writing are my own personal projects
If you have no real experience with TDD and you're hacking away at personal projects, I can see where you might not find TDD to be useful.
If you're experienced with it and working on a project that is going to be sizable and for use by others who might be paying to use it, TDD is indispensable.
I've seen a lot of fads come and go over the years. I've tried many of them out just to see how they fit my style of working. TDD is one of those paradigms that has withstood the test of time.
Much more important than code coverage and ability to make changes without breaking things as mentioned in the article, TDD forces you to think about your software components from the client perspective first. It's that discipline that I appreciate more than anything.
C.S. Degree in 1996, 20 years "professional" programmer and I never once thought TDD was helping my project. Every time I did it, it was because the boss told me I had to. Litmus test: every side project I did just for me, I never did TDD.
How many of those side projects ended up being 6 million lines of code maintained over 5 years? Because those are the kind of code bases I have in mind when I'm weigthing the pros and cons of TDD or other testing practices.
When reading Uncle Bob and others I have always got the feeling that the "goal" of the practices described is to have systems that can be maintained and extended for years by different people and teams. It never crossed my mind that Uncle Bob would recommend TDD for that tic-tac-toe I wrote to learn Buzzscript.
it's like i'm an artist. You can't tell an artist how to paint. I'm going to paint my best work when I'm allowed to choose my own easel and pallette and brushes. Let me throw some green paint on the canvas and make the trees how I wanna make the trees. Programming is more art than science.
Testing practices are orthogonal to the cathedral vs bazaar idea. Linux is a great example of this. Anyone can checkout the code and work on it, but upstreaming the changes requires coordination and approval with the community.
The lessons by Raymond are also given in the context of OSS. Unfortunately it would be extremely difficult to find a team of developers that are passionate about the kind of stuff that SAP is written to solve, for example.
The lessons of Robert Martin and others are about how to be a professional developer. When you are writing software for someone else to use and they pay you money to do it, it is your reaponsibility to think about quality, extensibility and maintainability. It is not your job to express your self.
My take is that the point of TDD being so rigid is that a professional should always follow best practices, not just when they feel like it or when it is easy or convenient to do so. But TDD is an ideal and sometimes, or maybe even most of the time, there are externalities that make you fall short of that ideal.
really good point about SAP! It's soooooo boring. That's the million dollar question - how to u inprise passion in a team of programmers all working for cash and not passion? I'd argue forcing TDD on your most brilliant artists is not the way. Or telling them that they are not "professional" if they don't follow what you consider to be "best practices" is a bad idea. That spark of true passion is worth it's weight in gold. Your best shot is to make an env where those sparks CAN happen vs. extinguish each little spark before it can grow.
Instead of test first, I often find myself doing "make it work, then write tests". It's all in the context of a single feature branch, so I have pretty good tests and coverage most of the time.
There is no perfect system or test suite. This is a reasonable 80/20.
[TDD] encourages a focus on sorting out detail to pass tests rather than looking at the program as a whole.
I have actually found the opposite to be true.
I have to make large refactorings to move things around to arrange the whole system so that each part can be tested without too much effort. To do this I have to view most things in terms of the interfaces they provide. On the test side, I have to write the test code so that the what the test does is strictly separated from the how the test does it, so that changing the system causes only minor changes to ripple to majority of the test code.
Based on this, it seems that programming with TDD is a distinct skill-set that requires significant effort to get reasonably good at, i.e., to be more productive than without TDD. I also have given it a try on medium-size projects and it does pay off in terms of simplicity of the design (I have to manage dependencies and decouple external systems and components quite heavily), low defect rates in production/qa, waaay less time spent in debugger, and high velocity (based on customer and product owner feedback at least).
However, the problem with TDD is that all of the above (tests decoupled from interface, interface decoupled from implementation, system decoupled from external systems, components decoupled from each other, design skills to recognize this, and refactoring skills to do this fast enough to remain productive) need to be done well enough at the same time. Otherwise the approach falls into pieces at some point.
To paraphrase Uncle Bob from some years ago: I have only been doing TDD for five years, so I am fairly new to it.. Half of the programmers have that much experience in programming in general, so the amount of time required to hone TDD and refactoring skills may not be there yet.
So maybe I am saying that you are not doing it right, but I don't really know. Maybe I am wasting my time writing tests, but anecdotally, I seem to enjoy extending and maintaining the TDD-based pieces of code more than the non-TDD pieces in our codebase.
I've been doing TDD for 4 years now, and I would totally agree that it takes a lot of time to hone the skills. My experiences of TDD on projects is quite similar to yours. However, it took me a year to simply really understand how to do TDD in any kind of sane way.
TDD is not something you can easily pick up in 6 months without a lot of mentoring and training from experienced TDD'ers.
Part of the problem is that there are not that many TDD codebases or TDD'ers around. Also, this is probably not something you can pick up while doing toy projects or school assignments. The benefits start to show in the 100 kloc and above magnitudes, and as there are so many ways to paint yourself into a corner with bad overall design, coupling, unmaintainable (or slow!) tests, chances are, you don't figure out all the necessary things yourself. On top of that, there is no time to learn this much in most dev jobs, so you are left to learn with hobby projects (which do not usually grow big enough).
223 comments
[ 4.6 ms ] story [ 242 ms ] thread"You are not doing it right! Take this 3-day course for a grand and buy these books. Also hire a TDD/Scrum/Agile coach for your team for a few months. There you go! (in Eric Cartman's voice)."
http://david.heinemeierhansson.com/2014/tdd-is-dead-long-liv...
Interesting. I've often found that the lack of tests leaves me absolutely terrified of making changes to large Ruby and JavaScript applications. I mean, if a test breaks, then I get a nice message right on the spot. But if I don't have tests, then the application itself breaks, and I may not find out until it's been put into production on a high-volume site or shipped to users.
Once an application crosses, say, 25,000 lines of code, it's hard to keep an entire program in my head, especially in a dynamic language and with multiple authors working on the code base. Under these conditions, large scale refactorings or framework upgrades can cause massive test failures, but the only alternative is to cause massive, unknown breakage.
I find tests to be extremely helpful for designing APIs. I write tests to enforce the contract my API is making with its consumers. Therefore, TDD forces me to consider the API from the user's perspective, rather than designing an API based on implementation details. That also helps to cut down on the cost of refactoring—unless I'm making an API-incompatible change, my tests should keep passing. This gives me more confidence as I'm refactoring, allowing me to make more sweeping changes.
I tend to write tests when I'm not confident in the implementation or when the tests are trivial to write. At least, that's how I do it for new developments. If you're releasing code that people are relying on, you should have a whole suite of user tests and regression tests.
But TDD, in my experience, gives a slightly different dynamic. Because you generated the code being motivated by the tests, there are a lot of unit tests that don't test functional units - tests that are essentially testing implementation decisions.
I've seen situations where that big refactoring would involve scrapping or rewriting tests, and that causes its own kind of architectural conservatism.
I ended up in a compromise position. I use a lot of tests, but I only do TDD in niche cases where it suits the problem I'm solving.
I've seen this happen to myself numerous times as well, but I don't think this is good argument against TDD.
If a tests fails because the implementation details change, that tells us that we wrote a bad test, not that TDD is bad. As I get better at writing tests, I am better able to catch these kinds of errors ahead of time. There are a ton of little techniques you can use and questions you can ask yourself about your code that can make your tests better.
Furthermore, test code is just as important as production code, and we should be just as ruthless in refactoring and peer-reviewing each others tests (within reason).
TDD is NOT suitable for every problem, this is true. But I think too many people give up on it because they see "the problems of TDD" which are actually just "the problems of writing bad tests".
Note: I'm not trying to target you sago, but merely making a general observation. I'm talking more about people who don't write tests even close to the time of writing the code, or don't write any tests at all because the their tests kept breaking for no good reason and were slowing them down.
I strongly disagree. The only thing that matters to how software works for an end user is its boundary. You simply can't do TDD by only testing the boundary.
You have to write a test before each unit of code, so you have to test those internal interfaces, the way units of code work. But that is exactly the stuff that should be allowed to change. It's a mistake to assume 'implementation details' only exist within a function. That stuff you can avoid testing, but the behavior of the function is itself an implementation detail for whatever is using that function.
I simply cannot imagine TDD for anything other than a toy example where you don't end up writing tests that depend on the software's implementation, and are therefore inertia to changing that.
> we should be just as ruthless in refactoring and peer-reviewing each others tests
I strongly agree. But this is rarely part of the TDD ethos. Saying "I refactored the code and threw away 100 tests." is likely to send TDD folks twitching, in my experience.
Now, safety equipment in a race car is something else entirely. Half of it keeps you from dying immediately on impact, sure, but the other half actually lets you go faster. The 5 point harness, for example, helps keep you from losing control of the vehicle during high G maneuvers, by keeping your torso in the seat and therefore your legs and arms in (roughly) the correct orientation to the steering wheel and the pedals.
You're ignoring the fact that tests aren't only to validate "end user functionality". They're also there to make life easier for the developer; to make both initial development and maintenance easier.
If I need a custom sort algorithm for my product, I'm going to write unit tests for it's implementation. I'm also going write functional tests to make sure the end user is seeing things sorted. However, those tests serve different purposes.
The unit tests allow me to write clean, simple tests that show exactly what the implementation is supposed to be doing for different inputs. That allows me to change the implementation of the sorting algorithm later and worry less about breaking the functionality.
The functional tests are (generally) not going to be as clean and obvious, nor cover every possible partition of inputs that the implementation could get.
Erm, no, that was my entire point!
I think my points align pretty well with what you were saying instead of against it.
I would start with writing an assertion about the end state of the sort. Then using syntactic substitution of the predicate calculi derive the program by assertions.
I'd then have a specification of the algorithm as a mathematical model from which I can check every possible execution for the entire domain on the sort. If I'm so inclined I would have a good basis for a proof.
The unit tests then become assertions on the implementation and don't have to verify correctness.
It would be awesome though if I could just generate the code directly from the specification!
It's easy to dream up algorithms and traditional unit tests for this (and easy to see how the large input space makes the approach suboptimal). I'd love to see how your approach would work.
I'm familiar with Coq and Prolog(one of which I think you're hinting at), but I'd reach for Perl or C over either one if I needed to write a trading program or generate a statement from my accounting software or something.
I think of it like this: nobody is going to care if I screw up the script to balance my accounts. If I write some software that messes up the balance of everyone's account in some bank then someone is going to care.
I don't think formal methods and rigor are the sole domain of government institutions, avionics control software, and industrial robotics. The public relies on the cloud and the applications that run on it and they have no recourse if their data is exposed or disappears.
Imagine if instead of crossing over a bridge you were stopped at a booth and asked to sign a wager stating that nobody is liable if the bridge collapses underneath you while you cross it. Rich or poor, regardless of how much you pay in taxes, you should be able to rely on that bridge being built to the highest standards we know how and someone should be liable if it falls down. You can rely on this in real life. Why do we not have this for software?
I'm not saying I write all of my software using formal methods and rigorous proofs. It's often good enough to sketch my idea on the back of a napkin when I'm building a tool shed. But nobody built a sky-scraper without doing a little math and being fairly rigorous in their thinking.
I'd be interested in knowing how, and what kind of thing you build. I can almost imagine it for an API or framework, where almost every implementation choice is on the boundary. But even then, you have to test private functions to do TDD, surely.
I work on real time stuff, and in that stuff, the user-facing functionality is hard to test, and usually not tested in a TDD kind of way. The TDD stuff is all internal.
I call this test concreting:
https://hitchtest.readthedocs.org/en/latest/glossary/test_co...
The way to avoid it is to either be damn sure that the API you've surrounded is tight, clean and unlikely to need to change much or to test at a higher level.
There shouldn't be any. Those are not useful at all.
Additional point is that invariant-based testing can easily be transformed into automated fuzzy tests using Quickcheck and its analogues, and this can find immense amount of otherwise hidden bugs.
That makes no sense at all.
> Your example is a perfect illustration
I didn't give an example.
> invariant-based testing can easily be transformed into automated fuzzy tests
'fuzz' not 'fuzzy'. True, but irrelevant to the point being argued.
Yes, rewriting tests is part of a refactor. You always have to be willing to do that or just scrapping outdated tests.
There are a couple things to consider that sometimes become taboo in orgs--but shouldn't.
The first is that you don't need to test everything. I recommend having a great set of those API-level black box tests as an acceptance suite, but at a component or unit level you should pick your battles. If you'd architect for DI -only- to test, that testing had better be pretty necessary, otherwise you introduced complication (and risk) without a lot of benefit. That risk can actually overwhelm any benefit of testing.
The second is that you should absolutely scrap tests of implementation details at the point that the implementation becomes a black box. Tests are like scaffolding during construction, and we don't leave the scaffolding up when the building is done--we do inspections of the building instead. The same mindset should be applied to testing. Different phases need different tests, and too many tests at too low a level of abstraction absolutely cause maintenance and flexibility issues if you're not willing to let them go.
But that's no reason to back off entirely. SOME things need to be unit- or component-tested SOMETIMES. Those things generally benefit from TDD.
It's all about making intelligent decisions. There's a sometimes-controversial movement in QA/test called "context-driven testing" which comes down to there are very few always-best practices, and strategy should be tailored to your problem and situation--not done by rote, book, or prior (read, your boss's or CTO's) assumptions of what's always right. Otherwise you tend to get something more onerous than useful.
The considerations around that approach apply at the low-level test tier as well.
This works remarkable well in practice and allows for large scale refactorings under the hood with little to no impact on the tests. We can also mock databases, memcached, redis and graylog on their respective http/tcp/udp level. This in turn means no custom build mocks which could break when refactoring. The tests itself also contain no logic, they are pretty much just chained method calls with data that should go in and an expected response that should come out, along with a specification of all external resource our API fetches during the request and their responses etc. Any unexpected outgoing HTTP request from our server will actually result in a test failure.
As for scaling this approach, from our experience it works quite well, especially when you have lots of complicated interactions with customer APIs during your requests since the flows are super quick to set up.
For our use case we have some pretty "fancy" deep-equals logic for nested structures and allow to specific fields as "to be ignored" in our test expectations.
When it comes to speed, the most important part is to cut out everything you don't need, for us this means that our testing framework completely throws away the internals of Node.js HTTP layer. We never create a single TCP socket in our tests, which gives an incredible speed up. As a bonus this also allows to test timeouts and low level HTTP errors in < 1ms, since we can just return the timeout directly from the low level APIs and Node.js will invoke the timeout event on the http client handle.
> Running "noir:mock" (noir) task > ...................................................... > 1048 passing (10s)
And yes, that really reads 10 seconds and yes I always get a bit bored when I have to run the tests for one of our Django backends, feels like an eternity... :)
...sadly that's still only for a single project x config x arch combination. I still need to play around with incremental linking options that appear off by default...
In the face of grossly changing requirements, I've never had much luck keeping E2E tests up and functioning properly. And people have a bad habit of investing more time and energy than a particular test is worth in trying to keep it working or porting it to the new requirements.
Unit tests are cheap. If the requirements change invalidates twenty of them, you just delete them and write new ones. Easy.
Gotta be careful there chief. In practice is usually refers to "your experience" (but might not be mine).
My experience is on the flip side: end-to-end tests are super slow due various reasons...
http://hitchtest.com
I took this approach on several different projects, but I figured that a lot of the boilerplate/infrastructural code that you need to actually write these kinds of tests is poor or simply not available.
For example, declaratively starting and running multiple services together, in parallel and at the right time (with service dependencies) and printing their logs out.
Or, mocking the forward passage of time. (click something -> move forward a week -> click something else).
Or, asynchronously 'listening' using epoll for emails on a mock SMTP server that logs out the emails it receives.
Selenium's good for the web interaction stuff, but you need much much more than that to be able to effectively test at this level.
I think this kind of testing would be much more widely used and effective if the tools available were up to scratch.
In our special case we have about 100 different endpoints all versioned and all dependent on multiple endpoints from the (rather badly documented) APIs of our customer. Most of the work our API does is spent combining / enriching the customers data and performing integration across the subresources. Setting up individual mocks for every single on of these complex requests flows manually is pretty much impossible at this scale.
So doing black box testing and enforcing a 100% test coverage (best for avoiding dead code) helps keeping us sane. In the end we don't care so much about how the implementation behind our HTTP response looks as long as we return the correct data in the end. The code itself still has to look good though :)
We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.
Typically run as a pre-build sanity check when you have remote integrations.
On some Labs projects we've written extensive request tests and stub services based on documentation, then handed those to the upstream service. Usually there is some angst at this point, being the first time that any kind of TDD suite has turned up to ask awkward questions.
Sadly, despite not working, they were still about 4 years ahead of their competitors.
Those kinds of "atrophied" tests can make a refactor considerably more painful than it should be.
(Disclaimer: I work on Newman as a part of my day job)
If you changed that functionality intentionally, then the other things it interacts with will require a change regardless. Furthermore, if it's that small of a change(as usually, the smallest changes are the ones that impact the most tests), it usually doesn't require any change to the test, aside from adding/subtracting a parameter or something.
Major functionality changes should carry weight. It's like saying that a motorcycle runs so much faster without a person on it- it defeats the purpose.
Think of it this way: pre-testing makes you afraid to change you mind because you throw out your tests, not post-testing makes you afraid to change your mind because you might break something. If you think changing your mind is good the path forward is clear.
I greatly-prefer the warm fuzziness of knowing that I have a large number of unit tests as a "safety net" to detect if my own thoughts on what is "correct" are in some way wrong.
An even bigger problem is that large scale refactorings can result in the tests themselves no longer being correct (which isn't the same thing as “not passing”), if you test at too fine a level of granularity. However, when you wrote these tests, you couldn't have possibly foreseen a large scale refactoring 18 months into the future, so how do you tell in advance what the right level of granularity for your tests is?
I agree. If I'm working on some part of the app that doesn't have any test coverage, I might add some high level tests before I start.
In my observation and feeling, the statement: " Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests."
is most likely due to the low quality and low test coverage of the exiting code base. So that the tests are fragile, seemingly hinders changes.
If things are started with TDD, I would be really surprised that tests hinders change.
Just because there are unit tests, doesn't mean they are covering all the code foremost, or that they are written properly to truly detect when things have changed for the worst.
Tests are not a replacement for actually testing and verifying your changes in whatever application gradient you're working in.
I too don't have the disposition to patiently wade through fixing tons of tons of broken tests over and over again. IF I have to I can do it, but I know better than to unreservedly trust my judgement about how things are progressing.
But what this has taught me is to stop trying to push water uphill. If testing is hard because of questionably coupled code, refactor it NOW instead of waiting for things to get bad. The refactoring almost always suggests new features we could add to the code (or makes me backpedal on pronouncements that certain things were 'impossible'), and the number and kind of tests that break is reduced.
I came across a quote recently from Bertrand Meyer (the Design By Contract guy), where he suggested that code for making decisions and code for acting on those decisions should be separated. I found myself nodding along to this because it was something I knew intuitively but had never articulated: Decisions without actions don't need mocks, and actions without decisions need at most one or two (often none). Decisions can be tested (possibly in parallel) with unit tests, and actions tested with functional tests. Then all your integration testing is that decisions lead to actions (basically that event dispatch works) and that catastrophic failures are handled gracefully. A proper testing pyramid.
Now I want to figure out which of his books or interviews this was in because I want to read the rest of what he had to say on the subject.
It's an excellent process to create stable and maintainable code, but it does not fit every bill.
But abandoning it completely on the grounds that it sometimes makes you write "bad software" is a bit weird to me, in fact, one of the main arguments for TDD is that it makes you write better code.
I found that it does make you write better code many times. So like all things, use when appropriate.
I guess the real hard thing is to determine when using TDD is appropriate.
You're points are all good, but your opening one especially.
The obsession that some developers have with methodological purity can be really puzzling sometimes.
Where I'm at: I work in research and engineering (in the gov't, on the borderline between industry and academia), and I can say that doing any sort of testing at all (either up front or later on) is an improvement over the untested, undocumented, unversion-controlled status quo that exists in a lot of cases.
TDD is super useful for simple algorithms. It gets harder once you get into more complex scenarios like multiple components working together. If TDD is not useful for some cases then don't use it there and use it where applicable. Or think about how you can make it work. It's not that hard.
For the first point, I'm reluctant to make large-scale changes to code which doesn't have lots of unit tests because the unit tests give me confidence in what the behaviour of the system should be.
Certainly the third point is something I try to bear in mind when doing TDD. I've found that having someone do a code review after a feature is complete gives someone the opportunity to come in from that high level and look at the program as a whole and check that your design makes sense.
I've found it's pretty useful to go back and add tests as you accumulate users, though (or convince an exec that your project is Too Big To Fail in the corporate world). You're capturing your accumulated knowledge of the problem domain in executable form, so that when you try an alternate solution, you won't suddenly find out - the hard way - that it doesn't consider some case you fixed a couple years ago.
My initial comment was meant to be a statement about the relative chance that you're solving the wrong problem vs. solving the problem wrong. Prototypes help you identify that you're solving the wrong problem, unit tests help you identify if you're solving the problem wrong. In the beginning of a project, you are much, much more likely to be solving the wrong problem than solving the problem wrong. Go ensure that the system works end-to-end and solves the user's needs before you make it bulletproof.
For this kind of coding, TDD makes no sense whatsoever. The 'specs' are as fluid as the code and having confidence in the code isn't that important.
This is entirely different to creating production hardened systems with very clear specs. If you don't do TDD on that, you're an idiot.
What steps would you take?
So you write the code; expose it to a preference function / oracle, iterate, and hill-climb towards a local optimum. The revealed preferences may give enough information to make a creative leap off the local hill onto a large hill elsewhere - but it may not.
After a solution has been found, then you can write tests.
Is this preference function implemented on a computer, or is it just a person giving you seemingly random answers in response to whatever you tell them?
> People "know it when they see it" without necessarily being able to define it.
That path leads to hell.
> After a solution has been found, then you can write tests.
what exactly would you test? Tests make sense when you want to check that your implementation conforms to a certain set of logical rules, but where exactly is the logic in the process you outlined?
Seemingly random answers, which are not functionally pure (aka ask the same question, get the same answer) are par for the course
> That path leads to hell.
That's why they pay us the big bucks.
If people could define their problems well enough without iterative design, waterfall development would be a success story and all software would be outsourced to the lowest bidder.
The point of spiking is that you don't know enough to TDD. You are outside the sweetspot where TDD is tractable.
The key to spiking is that once you come with a plausible approach, or have a conceptual breakthrough, you stop, backtrack and then switch back to TDD.
Like everything else in mature software engineering, agile or classical, it requires sustained discipline.
(They'll try to recruit you, but TANSTAAFL).
That's the beauty of TDD. It forces you to sit and work out, "What is this code supposed to do from the client perspective?"
It battles the all-too-prevalent developer inclination to just start running with implementation without even taking the time to understand what problem is being solved.
If the same situation is tested in 2 different ways, a bug in one test might cause the test to fail to correctly handle all case, but the second test might still catch.
Maybe auto-generation of test cases would be a better technology?
A feature/e2e test typically provides the most effective kind of fault detection, because it combines all the components of the real system with no stubs or mocks. But if it shows a fault, then typically that fault is hard to identify, because it wasn't previously driven out by a more detailed integration or unit test.
Contrariwise, integration or unit tests will typically be best suited to isolating faults, but typically those faults are limited to the class of things you included in your tests.
This is why we have "test pyramids": a handful of slow, brittle feature tests at the top, then an increasing volume of faster, less-brittle integration and unit tests.
TDD is almost orthogonal to the testing pyramid, with one difference from non-TDD tests. In TDD each line of code was, ideally, driven out by a test or tests. Fault detection is increased, because each line was written in response to a manufactured (test-first) fault.
In some situations I have clear view of what I need to build, and how that should work. TDD is great in that case - write a test for the expected behaviour, make it pass, refactor, rinse and repeat. The element I think a lot of people miss when doing this is higher level integration tests that ensure everything works together, because that's the hard bit, but its also essential.
Other situations you're still feeling out the problem space and don't necessarily know exactly what the solution is. There's an argument that in those cases you should find a solution with some exploratory development then throw it all out and do it with TDD. If I've got the time that's probably true, it'll result in better code simply through designing it the second time with the insight provided by the first pass, but often that just isn't viable. Deadlines loom, there's a bug that needs fixing elsewhere, and I've got two hours until the end of the week and a meal date with my wife.
Finally there's the times when tests just aren't needed, or they don't offer a decent return on the effort that will be required to make them work. I'm thinking GUIs, integration testing HTTP requests to other services, and intentionally non-deterministic code. Those cases certainly can be tested, but it often results in a much more abstract design than would otherwise be called for, and brittle tests. Brittle tests mean that eventually you stop paying attention to the test suite failing because its probably just a GUI test again, and that eventually leads to nasty bugs making it into production.
One thing I'll directly say on the article is that I found his opinion that its hard to write tests for handling bad data. That's almost the easiest thing to test, especially if you're finding bugs due to bad data in the real world - you take the bad data that caused the bug, reduce it down to a test case, then make the test pass. That process has been a huge boon in developing a data ingestion system for an e-commerce platform to import data from partner's websites as its simply a case of dumping the HTTP response to a file and then writing a test against it rather than having to hit the partner's website constantly.
Hear, hear! Exactly the same here, for the same reasons as you and the OP mention.
And observed fom a distance, it's always the same universal principle: purist behaviour (in the sense of almost religous beliefs that something is a strict rule, sentence starting with Always etc you get the point) in programing or to a further extent, in life, is nearly always wrong, period. No matter what the rule is, you can pretty much always find yourself in a situation where the rule is not the best choice.
That's not really TDD anymore though - post-hoc testing of bad data is always going to be orders of magnitude easier because you have the bad data and you know what broke.
This is exactly why I don't do TDD. I prefer to think about the design and implement features in non-distracting way. After I have it shaped, I add tests to avoid regressions.
Where I do write tests first is when I need to reproduce a bug, so I have the test, again, to prevent regressions.
It's extremely tricky to avoid being threatening as a leader. Your position gives you power nobody else believes you deserve. Even when, sometimes, you do. Not saying I deserve it, but there are those that I've worked with that did, and I only noticed in hindsight. F'ing limbic system.
More realistic, concrete examples are harder to describe because they involve an interaction over time between a somewhat vague problem description and exploration of a solution state space, where the abstractions chosen are fluid and slide around before they get into a good shape.
I, for one, tend to write code from the bottom up, i.e. creating hypothetical abstractions, small tools etc. and start composing them to solve the next problem up the abstraction stack. Then, when I find it doesn't fit quite right, I adjust, freely throwing away abstractions, rewriting them, reshaping, until the level of my tools fits the problem better. I gradually build up my abstraction level until I can move problem domain level mountains with little effort. Doing this in the form of tests just doesn't work (for me).
Writing code from the top down isn't the right answer either; not all people think that way, and besides, it can lead to solutions with very ugly implementations - the grain of the wood should influence the design of the house, if you will.
Kent Beck in his book doesn't recommend top-down, or bottom-up, but rather from known-to-unknown. Start with what you know, and work toward what you don't know.
Edit: The books I refer to are Test Driven Development: By Example - Kent Beck, and Clean Code - Robert Martin
We don't use e.g. units-of-measure types [1] because they remove duplication from our code; if anything, they add duplication. They do however clarify our thinking with help from the type checker.
I think a lot in terms of flows / pipes / functional transforms. So I tend to try and express problems in that shape, because I have a lot of mental tools that I can apply, and I know they're extremely easy to test in isolation. Creating a pipe-like thing means reducing it to a simple common push or pull stream pattern. But it's not duplication I'm removing here; I'm actively introducing an abstraction because it has proven power for creating good software and solving problems.
I try and create few, minimal abstractions that can be applied widely. Take a cue from functional languages and split out types from algorithms; if you make your types more general, you increase the reusability of your algorithms. Classical OO design tends to create a lot of types that are specific to the domain model. I happen to think that OO designs are usually poor; they tend to have a high code complexity to implemented logic ratio, and require awkward composition that leaves details hanging out.
I spent 20 years writing OO software and was a big fan, especially in the early 2000s. I still work in OO languages, but most OO code I read makes me wretch now.
I don't recognize either Kent Beck or Bob Martin as particularly noteworthy for good architectural design (Beck is on the right path with agile though). In fact, I blame TDD for Java-itis: proliferation of single-implementation interfaces (has there ever been a worse idea more often propagated by cargo culters?), poorly abstracted object graphs, leaky implementation abstractions, and more.
> Start with what you know, and work toward what you don't know
I've been programming for more than 25 years. A lot of stuff has changed in that time; what hasn't changed is patterns of abstraction. So I start out with abstractions, chosen from experience.
Read Norvig on TDD [2] - his experience matches mine. If you know a lot of software tools (i.e. abstractions, algorithms, approaches), you can apply them to a problem. TDD is a poor tool for creating new tools, though. And if you rely on code duplication for creating a tool, well, you're going to have a bad time with hard problems.
[1] https://blogs.msdn.microsoft.com/andrewkennedy/2008/08/29/un...
[2] http://pindancing.blogspot.co.uk/2009/09/sudoku-in-coders-at...
http://www.artima.com/suiterunner/private.html
I strongly disagree with this statement. The best design for your program is always the one that's easiest to test; the one that's modular, the one that separates out dependencies, the one where methods are as atomic as possible.
If you find yourself wanting to write code that is hard to test, then you are approaching the problem (and/or the solution) the wrong way.
> The ‘purist’ approach here, of course, is that you design data validation checks so that you never have to process bad data. But the reality is that it’s often hard to specify what ‘correct data’ means
Again I disagree. It should be really clear what correct data means at a low level. If you can't, then you haven't fully fleshed out your design, so yeah it's gonna be impossible for you to test it.
I'm far from a TDD zealot. But on my team we adopted writing unit tests for everything in the last 6 months and it has been night and day. It is incredibly useful and the resulting code is so much better.
I think a lot of the author's problems stem from attempting to do this alone and not having someone else to guide him through how to tackle things that he's having trouble with. Plus, frankly, he seems to just have a defeatist attitude about the whole thing.
Tests are not about "code coverage", nor about establishing the exact sequence of things in stone. Tests are about fixing invariants.
When a new project starts, I only know about 10-15% things for sure, and those are exactly which will go in tests, before writing any new code. I don't worry about some things in my code are not yet covered by tests; I don't know yet how they will turn out, so I can't write any meaningful invariants.
In my experience, useful tests are much higher-level than TDD guys prefer. They routinely fix invariants for the entire system / subproject, not assuring coverage of every method in class (some crazy folks are even testing getters and setters — why?)
Well, I can see the logic if your getters and setters are hiding more activity than simply retrieving/setting the value of a private field, which is the point of having separate getters/setters at all.
If you imagine a getFullName() / setFullName(name) pair, for example, that actually reads from/writes to two different private fields for first and last name (leaving aside middle names, internationalisation, etc), then there's some minimal logic there that you might want to test.
In a duck-typed language, when you're trying to ensure a class obeys an implicit interface, it may also have value.
Apart from that, for vanilla getters/setters, it's a little pointless.
EDIT: typo
* fixing bugs - reproducing the bug in a test case both confirms you're fixing the thing, and acts as a regression test
* defining a protocol - where you need to glue two things together, e.g. a front end UI and a back end controller, or a model shared between different modules
It's a lot weaker for design. Test-first tends to encourage overly open to extension abstractions, because you need to make things visible to tests and make components replaceable by mocks. In the early stages, the weight of updating tests makes the design overly awkward to change. And early on in the process is exactly the wrong time to be creating your general abstractions - that's when you have the least amount of information about what will make for the best abstraction.
You still need to back-fill tests after good abstractions have been chosen, of course. Tests are great; test-first, specifically, isn't always best.
Summary of how they work: you say "this program takes a 32 bit signed int and a string" and the testing framework will throw it a heap of random ints and strings, some of which match the sorts of classic curve balls it might encounter (negative numbers, int max and min values, strings that are very long, empty strings, strings with \0 in them, strings that don't parse as valid unicode, "Robert');DROP TABLE Students; --", and so on.)
By focusing on the specification, you can ensure that design decisions are made by the coder are fit for purpose.
I also think many tests would be better written as code contracts, you mainly want to ensure the inputs and outputs are valid, code contracts focus on this.
Is he arguing that simply trusting yourself not to make mistakes is a sufficient guarantor of quality? Ian Sommerville is the author of a famous textbook on software engineering, so it would be surprising if he was.
TDD is actually much more difficult in practice than people realise. I read Kent Beck's book and thought it sounded like utter horseshit. I tried doing it myself and decided it was definitely horseshit.
Then I came to work at Pivotal Labs. Now I am distrustful of code that was written before tests.
As for the argument that TDD distracts from the big picture, this is like saying that indicator signals and mirror checks distract from driving. Sure, when you are learning to drive, you feel so overwhelmed by attending to all the little details that you struggle to drive at all. You become unable to focus on your navigation.
After a while you learn the skill and it becomes automatic. TDD is such a skill.
And what do you get with regards to specs? It seems to me the methodology fits how requirements are derived.
The bread and butter of Labs is teaching people pair programming and TDD. So I was taught those things, and I'm still learning.
It takes a few passes through the whole cycle of story->feature test->integration test->unit test for things to begin to click, for that way of thinking to become habitual. It is uncomfortable and frustrating for some time, as Ian Sommerville found. In fact, because most code is not written test-first, it is difficult to test. So all that keeps you to TDD is habit and, to some degree, help from another engineer.
> And what do you get with regards to specs? It seems to me the methodology fits how requirements are derived.
The most popular story-writing style I have seen so far is the As A/I Want/So I; Given/When/Then. Usually some acceptance criteria are supplied. To varying degrees of fidelity that the feature tests will typically follow Given/When/Then.
In our weekly Iteration Planning Meetings engineers are expected to pipe up if there are useful ways to decompose stories into smaller stories, which makes the overall situation more manageable.
The way to make testing pleasant is to extract more and more behavior into units with clear boundaries... realizing how to do this was a major event in my programming career, and I attribute the insight partly to doing TDD.
I don't agree with some posters who say that TDD encourages overly generalized design. Sure, it encourages some form of dependency injection... but mostly, it just encourages the creation of coherent and loosely coupled units, which is a universally lauded best practice.
It is hard to say for sure, because the author doesn't give any specifics but wording like:
"You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests."
make me think the author wasn't actually making unit tests, instead they were likely end-to-end tests, or partial ETE tests that were running inside a unit test framework. I have had many disagreements with other developers that just because your test runs inside a unit testing framework doesn't actually make your test a "unit" test.
As with everything there are costs and benefits!
A lot also depends on the quality of your tests. If the team doesn't care about test quality or doesn't know how to write good tests, you can easily get these big blobs of incomprehensible and tedious tests...
And certainly TDD is harder as you approach the GUI, you want to test in vague ways which don't break with every change. If you thoroughly test all of the underlying behavior, implementing a GUI is typically incredibly trivial because everything beneath it is known (and proven) to work. Most of the article is not related to the GUI.
> ...it doesn’t work very well for a major class of program problems – those caused by unexpected data.
This is a hollow argument. Regardless of development methodology, if unexpected data isn't considered at all it could have all kinds of side effects.
Regarding conservatism with breaking tests (many tests failing for one change), it's likely the result of a structural problem within the application if it's an intimidating number of failures.
> It is easier to test some program designs than others. Sometimes, the best design is one that’s hard to test so you are more reluctant to take this approach because you know that you’ll spend a lot more time designing and writing tests (which I, for one, quite a boring thing to do)
Not sure how this applies to TDD, if you're writing tests first you aren't deeply concerned with designing tests because you're imagining what the interface for well-designed code would be, and then you write it. It frequently sounds like the author jumps into writing tests without any forethought.
> In my experience, lots of program failures arise because the data being processed is not what’s expected by the programmer. It’s really hard to write ‘bad data’ tests that accurately reflect the real bad data you will have to process because you have to be a domain expert to understand the data.
If you don't understand the variety of inputs, how can you possibly validate them? Programmers should have some domain understanding, certainly program inputs fall within that realm.
> Think-first rather than test-first is the way to go.
I agree; but step 2 should be testing in my opinion. Test first is just the first tangible work product, it isn't a ban on thinking.
I deliberately decided to experiment with test-first development a few months ago.
and
the programs I am writing are my own personal projects
If you have no real experience with TDD and you're hacking away at personal projects, I can see where you might not find TDD to be useful.
If you're experienced with it and working on a project that is going to be sizable and for use by others who might be paying to use it, TDD is indispensable.
I've seen a lot of fads come and go over the years. I've tried many of them out just to see how they fit my style of working. TDD is one of those paradigms that has withstood the test of time.
Much more important than code coverage and ability to make changes without breaking things as mentioned in the article, TDD forces you to think about your software components from the client perspective first. It's that discipline that I appreciate more than anything.
When reading Uncle Bob and others I have always got the feeling that the "goal" of the practices described is to have systems that can be maintained and extended for years by different people and teams. It never crossed my mind that Uncle Bob would recommend TDD for that tic-tac-toe I wrote to learn Buzzscript.
How many projects at all end up that big?
How many of those end up falling under their own weight?
How many of those failures are attributable to lack of TDD?
How many of the succeeding ones succeeded on account of TDD? Or in spite of TDD?
All I've seen so far is anecdotal evidence, which is no evidence at all.
The lessons by Raymond are also given in the context of OSS. Unfortunately it would be extremely difficult to find a team of developers that are passionate about the kind of stuff that SAP is written to solve, for example.
The lessons of Robert Martin and others are about how to be a professional developer. When you are writing software for someone else to use and they pay you money to do it, it is your reaponsibility to think about quality, extensibility and maintainability. It is not your job to express your self.
My take is that the point of TDD being so rigid is that a professional should always follow best practices, not just when they feel like it or when it is easy or convenient to do so. But TDD is an ideal and sometimes, or maybe even most of the time, there are externalities that make you fall short of that ideal.
When George Lucas was given unlimited time and money, he produced inferior films.
McCartney and Lennon's solo efforts don't stand up to their collaborative works.
Elvis eventually became fat, gaudy Elvis.
There is no perfect system or test suite. This is a reasonable 80/20.
I have actually found the opposite to be true.
I have to make large refactorings to move things around to arrange the whole system so that each part can be tested without too much effort. To do this I have to view most things in terms of the interfaces they provide. On the test side, I have to write the test code so that the what the test does is strictly separated from the how the test does it, so that changing the system causes only minor changes to ripple to majority of the test code.
Based on this, it seems that programming with TDD is a distinct skill-set that requires significant effort to get reasonably good at, i.e., to be more productive than without TDD. I also have given it a try on medium-size projects and it does pay off in terms of simplicity of the design (I have to manage dependencies and decouple external systems and components quite heavily), low defect rates in production/qa, waaay less time spent in debugger, and high velocity (based on customer and product owner feedback at least).
However, the problem with TDD is that all of the above (tests decoupled from interface, interface decoupled from implementation, system decoupled from external systems, components decoupled from each other, design skills to recognize this, and refactoring skills to do this fast enough to remain productive) need to be done well enough at the same time. Otherwise the approach falls into pieces at some point.
To paraphrase Uncle Bob from some years ago: I have only been doing TDD for five years, so I am fairly new to it.. Half of the programmers have that much experience in programming in general, so the amount of time required to hone TDD and refactoring skills may not be there yet.
So maybe I am saying that you are not doing it right, but I don't really know. Maybe I am wasting my time writing tests, but anecdotally, I seem to enjoy extending and maintaining the TDD-based pieces of code more than the non-TDD pieces in our codebase.
I've been doing TDD for 4 years now, and I would totally agree that it takes a lot of time to hone the skills. My experiences of TDD on projects is quite similar to yours. However, it took me a year to simply really understand how to do TDD in any kind of sane way.
TDD is not something you can easily pick up in 6 months without a lot of mentoring and training from experienced TDD'ers.