Not much to add except that when j tried TDD, I had to have this mentality of typing out code as fast as I could, going back to the test, fixing stuff, back to code, learn the mistakes, etc.
While my dev approach is more about think, think, think harder, code, compiler barely complains, fix small issues, write test, done.
TDD felt very uncomfortable because the thinking part didn't seem to be emphasized much. It's like keep going, you iterate again real soon in this code, now again, and again.
Maybe I got it wrong, I don't know. But I share the author's frustration.
First thing: they do not define which kind of TDD they are talking about. Second thing: without really good test coverage any kind of larger refactoring is gambling with reliability and will bite you at some point.
I have never actually seen somebody writing all of the tests before implementing the code even if they claimed doing TDD. Parts of tests maybe but I think that things rarely go the way the definition is spelled out.
> Don't seem fair to call out author on not writing tests because they never said that.
Fair point. I have written my comment halfway into the article, the author makes a point about tests at the end.
> I have never actually seen somebody writing all of the tests before implementing the code even if they claimed doing TDD.
I had an architect who did exactly this. It was awesome, one could just look at his code and see where it would fail because it didn't occur to him to either test for that case or actually think what the code does. His only explanation would be "but it passes the tests". Later he was made into CTO.
In a way, not testing for edge cases that are found during development is quite similar to commenting on article one hasn't fully read.
Right, I write tests for existing functionality. And I decide what test to write after I write code. In the spirit of this article, I would assert that a simple two line integration test could cover a large portion of code depending on how many parts it requires to work together correctly.
This wiki has great content about the different types of TDD. Reading it helped me get useful insights about my approach and my gripes with TDD. In particular it's interesting how different approaches to testing influence heavily the design of your code.
True. I just run into these situation too much because I functional style code where there is very little internal state and try to use pure functions as much as possible. This narrows down the code that has to be written on a more testable way.
In college we were always taught to do TDD on large projects, and everybody I know does TDD(at least here in France). I can even think of working on a project that's not TDD. You need to look up Agile, Clean code and Xtreme Programming ;-)
This comment in the original article is really to the point:
I think one of the biggest values of TDD is it teaches developers how to write unit testable code, which, if you are new to unit testing, can be quite hard to get your head around.
As for its usefulness in ‘real-life’, I’m going to answer with a solid ‘it depends’.
Anyone who dismisses it is missing out on some potential big advantages, anyone who is a 100% TDD zealot is probably either wasting time or lacks extensive real world experiance.
HN loves shallow opinion pieces, especially when they're contrarian. They give us a chance to wade into the comments and demonstrate how smart we are, either by agreeing or disagreeing!
The experiences of beginners are as valid as anybody else's.
It creates an unfriendly culture if people are told they can't talk about their frustrations and challenges without having x years of experience first.
The experience of a beginner with a certain tool is quite important. E.g, if Linux is not user friendly to newbies something could be done about that.
But some tools require expertise and a beginner might not get the entire picture yet. E.g, if someone with 1 year of physics experience claims Quantum Mechanics to be dumb, is that a valid opinion?
>The experiences of beginners are as valid as anybody else's.
In certain situations they may be even more valid. But when it comes to software processes you really need to have lived and breathed them in order to mount a credible argument.
>It creates an unfriendly culture if people are told they can't talk about their frustrations and challenges without having x years of experience first.
Nobody is denying this person the right to talk about their frustrations.
Nothing wrong, but also nothing worth discussing. It doesn’t qualify as a good research or an opinion of a subject matter expert - just few banalities in a popular genre of provokative tech blog post.
Its always like this. Last month the article was about how OOP was useless and a waste of billions of dollars, yet the author had like 2 years of experience as a Php developer (in 2019?). It didn't stop 2 of my sub par coworkers from sharing the article and acting like the author had any creds or experience at all to be saying such things. Generally I never read blog posts from no name people claiming industry standards should be destroyed. Its just a pointless exercise.
He's right, but also wrong. It really depends on what you are developing.
TDD is awesome when fixing bugs (as opposed to writing features), when writing straight up business logic (especially when there are a lot of edge cases) and when you know exactly how something is going to work.
It's really bad when doing anything that requires exploration and research (which is the example in the article).
It is still useful, just a different kind of test.
High level quality/performance check, rather than testing functions. Produces values, not pass fail. Not really a unit test, an automated integration test of a module.
Simple test run to detect crashes or utter failures in CI fashion helps with speed of development too. Also not a unit test.
Once we had a research solution and were setting up performance improvements, we have put regression tests in place for the replaced functions and the whole system, derived from earlier module quality tests. Added a few well chosen property based unit tests.
This has caught edge cases in the fast implementations.
We do the same at work. No unit tests, only integration and regression tests. See metamorphic tests (comparing outputs for mutated inputs).
One thing I've been trying out recently is adding asserts to code (design by contract): encode function preconditions, postconditions, invariants.
Now there are many more ways to "crash" the system while testing. The gist of it is: if you know that there is a bug if a particular condition happens, add an assert encoding this. Now all code that exercises the code with the assert automatically become tests.
I've found preconditions to be more useful than the other properties.
No offence but I'm amazed you weren't using assertions (and assertions as pre/post conditions, invars) from the start. They work so well. And true, you get many new ways to blow up the code; all to good.
(Edit: pre.post/assertions/invars etc are what I've used in my own code. There's little push in most places I've worked for any testing, never mind anything systematic like this).
You're absolutely right. Assertions are so easy to use and there's no barrier to start using them. You can start small, one assertion at a time, no large refactorings required.
For anyone reading this, I originally was taught assertions etc. at university but really got to use them after reading this ppt presentation by Tony Hoare (PDF here. Only took 1/2 hour to find, thanks for getting crappier, google).
Honestly, it's short, please everyone give this 10 minutes of your time. It was one of the main things that started my move from rando programmer to something closer to engineering practice. I just wish companies I worked for gave a damn for it.
He said that too, and sometimes is better to write the tests after you have the functionality finished and working because you know exactly what you expect and you add the tests to prevent feature updates to cause unexpected changes.
IMO TDD works if your input and output is clear from the start and the way to get the output is convoluted enough that you need the test.2
I work for a Danish municipality and we buy quite a lot of development from various software houses. Being the public sector we track and benchmark almost everything, and we actually have a dataset on automated testing that’s been running for two decades.
It’s hard to use the data, because we’re comparing different projects, teams and suppliers but our data shows no advantage in choosing the companies that are very test-focused.
They are often slower, more expensive but have the same amount of incident reports as the companies which tests less or doesn’t do automated test at all.
Companies that sell themselves on TDD (among other things) are part of our dataset. Maybe “automated tests” isn’t the exact literal description of that, but I didn’t mean it as such.
One question: are they somehow more efficient in answering to an incident report?
My conjecture: I am not so surprised at these having the same % of failures - after all the TDD will not magically lead everyone to think of all the possible ramifications; you write tests for the case that you can "imagine" just like a non-TDD developer will implement logic to cover cases they can "imagine".
But on the other hand, if after fixing the bug you need to prove that you haven't caused some other side effect, then having a full test suite in place should be more efficient than retest everything again (or less risky than "ok, fixing module W should not impact X, Y and Z, so we test X only and ship...").
I guess these are good points. The last line especially.
Yet it's obvious to me that correctness and stability should be features, and more fundamental than all others. It's not as if a non-extreme amounts of testing is all that expensive.
That's an interesting dataset and it aligns with my gut reaction: TDD (and testing in general) are ideal for bug fixes and code refactoring, which would be less useful when the software is going to be sold at a specific point in time (and therefore frozen or mostly frozen).
Interesting. My intuitions (as a developer of 25 years experience) are that testing can be a big tax on development velocity; perhaps 60% of time is spent on writing tests, especially in a more complicated code base where the choice is between tests with lots of mocks (=> brittle against code change), or tests with lots of data setup (=> brittle against schema change), where the tests are more verbose than the code under test.
I also think testing compensates for communication inefficiencies when developing with a larger team with developers with less tenure and experience. I'm not sure this tax is avoidable as you try and scale up organizationally.
Code architecture also has a significant effect on the cost of writing tests, and TDD was originally propounded as a way to ensure that code is testable, but with object-orientation it tends instead to lead towards dependency injection and pluggable interfaces purely for testing rather than reconfiguration, and you end up with overly abstract software.
More functional code, and architecture which increasingly represents control flow as data flow, is a better direction to aim in, IMO. That argues towards streaming and messaging techniques more than interfaces and APIs.
TDD is great for fixing bugs, less great for software design. I think the key reason is that the best design is in tension; it's a solution being applied to several problems, and triangulating that point starting from a single problem is both (a) further away from the solution, and (b) unlikely to be close to the final design unless you were really lucking in choosing the test problem.
That hasn’t been my experience at all. Test-oriented firms and test-oriented projects have had far fewer incidents and the incidents they have had have been resolved much faster. The data has been so conclusive that our exexutives are mandating that we test. Maybe the difference is automation? Automating the tests is key.
This may hold some of the answer. IMO, tests are basically automated debugging, that you can run as many times as you want. To take full advantage, you should run them basically whenever you can, in the background of editing, forced to succeed upon merge, etc.
I like this approach and it was used heavily in a previous role by both the devops and development teams in a company I worked for.
However it can come with a cost if the tests are not targeted enough or there are simply too many tests to run within a reasonable period. At one stage tests were taking 20-30 minutes to run, so you could be waiting an hour or more sometimes to find out whether the change you pushed went green or red. Often this meant people pushing before their lunchbreak or at the end of the day and effectively leaving the build broken for many hours, with others unable to push unless they included a fix for someone else's breakage.
I see a lot of teams/projects where the tests take 20 to 30 minutes. Like that's the threshold where you can just barely get anything done, so things grow to that size, development velocity grinds nearly to a halt, and they stop growing. So everything you see is about that size.
Personally I need tests to run in a few seconds. I work on a large project that does it so it's definitely possible.
I am lucky enough in my current role to have a lab environment that is almost 1:1 with our production environment, so we use testinfra to test all our VM's in parallel, with tests tailored for each VM to the task it performs (database, application, load balancer, DNS, LDAP, SMTP etc). We can run tests on an entire environment in under 2 minutes.
Far more holistic than the 20-30 minutes I described in that previous role, where the whole set of tests was run in a single vagrant VM, usually recycled to save time on spinning up a new VM for each run.
There's a lot of great stuff you can learn from devops, but there also seems to be a large amount of cargo culting where people have cut & pasted certain solutions from a blog or something without understanding how to adapt it to their environment and make it work properly for them.
You are self-selecting for companies that would be willing to go through the bureaucracy to be vendors for you. Consequently, the type of people these companies would hire may not be of high quality. This may or may not be true for you, but is true in my experience.
So the type of customer you are attracts those type of devs.
Do you do any verification on how test focused they are? IME automated testing is like agile, the companies that talk the most are awful at the execution.
I'd completely believe that the automated testing companies were slower and more expensive, but I'd be inclined to blame the quality of the tests themselves. I've seen so many code bases where the tests slow down development and rarely catch errors, where you'll often have to spend more time decoding what the tests were supposed to be testing than the actual development. Even otherwise good developers can see testing as a necessary compliance item and will write awful tests just to get coverage up.
In your dataset are there any standout companies with low incident reports which do/don't have automated tests?
> He's right, but also wrong. It really depends on what you are developing.
The more useful thing to notice beyond TDD is how this is true of almost every development model and programming pattern (including the patterns built into languages that pretend they are not) - development processes and programming patterns are almost never generally applicable - The problems come when people who don't understand this attempt to apply them generally and blindly, e.g the classic go a read a patterns book or development strategy book and now everything is a nail for their preferred choice of hammer.
I'm not being defensive of TDD though, just because it's subjectively useful does not make it equal, to me TDD looks to have very narrow utility in general for the vast majority of projects it's going to get in the way.
One thing to consider I think is that TDD isn't just about the tests. It's also about architecting the software into small functions and lightweight classes, which are easy to write concise tests for and then implement incrementally. To TDD advocates, this resulting characteristic of the code is a valuable goal in itself, even just as a side effect of TDD.
Right, so as the author suggests, maybe TDD's real benefit is that it makes you think and plan a bit more before jumping into coding.
Therefore, we should have MDD - miniplan-driven-development: make a mini plan for the feature being developed, not so dissimilarly from what you would do with the first pass of TDD. The difference is, once you start implementing that plan and realize the problem is much different, you don't have to go back and keep modifying your original miniplan. You just do what you would normally do.
What you gain is probably a partial improvement in your development process without much cost.
My experience is that the resulting architecture will sometimes end up with characteristics that makes it easy to test but more complex, exposes things it should not and too many layers of indirection.
I do like automated tests (unit and integration) and often TDD is appropriate. However tests are not the end goal of an architecture. The code should be maintainable, easy to follow and simple to reduce number of bugs. If TDD starts to change that by adding layers of indirection or changing the purpose of something just so you can do a fast unit test then something is wrong.
> It's really bad when doing anything that requires exploration and research
I wouldn't say that, I have an counterexample ([1]):
It is clear what the refactoring (that is to be implemented) should do and how to state that ([2]), but it usually requires exploration and research how to work with the AST and which methods to use.
I would even say that TDD is awesome for exploring, researching, for getting a feeling of the problem's context, what to do and what the caveats are, as long as neither of that is required for stating the test itself.
Having tests first, you can simply start the debugger, try things out and with hot reloading you can write the code while running the specific test that you want to concentrate on.
Visualization tools like [3] can greatly help exploring how to solve and implement the current test while you debug it.
TDD can work if the tests are integration tests and each test case covers an entire feature (one which can be clearly described using non-technical language). Integration tests only lock down the high level interfaces and behaviors so they allow you to keep reworking the guts of the system without ever having to rewrite the test cases.
Unit tests are mostly only good in the very final stages of the project when you have a fully working, essentially perfect, bug-free system and you just want to lock it all down so that nobody can make even the tiniest changes to the code without breaking all the tests and becoming insecure about their abilities in the face of such a remarkable code base.
If you're writing unit tests for poorly architected code, you're essentially locking down this horrible architecture and forcing all future developers to find complex ways to work around it instead of allowing them to confidently make the necessary restructurings to allow them to cleanly build on top (this kind of "I'm not breaking any features" confidence can only come from integration tests).
When it comes to complex systems, writing unit tests too early in the project is a bad idea; especially when combined with an agile development methodology.
It's like building a skyscraper over several years and perfectly analysing every steel rod, every brick and every pane of glass as it is added to the structure but never actually stepping back to check that the structure as a whole is actually pointing upwards.
Very well put. It is a great paradigm for problems that you understand very well.
But I think we can also define it more inclusive and still be fine: TDD is a good paradigm for programming against well defined APIs.
For example if you are handed an API specification then locking down your interface(s) with tests in advance can be very helpful.
What is even more helpful is to have a higher level language or abstraction that encodes the specification in detail and generates the tests for you (Clojure spec comes to mind).
I don't typically tdd, but I'll say TDD is also awesome when you are in a crunch and working on some feature that has, say more than one dependency and more than say four corner cases and you are all amuddle about which one you should do first. Especially, london-style? Is it? Where you write out your trivial stub implementation first, helps me sort it out and not worry about thinking of everything.
Everything has a cost. TDD (and Refactoring) increases the cost of implementation (linearly), but they greatly reduce the cost of change. Without TDD (and refactoring) the cost of change might become exponential and impossible to catch up to.
Assuming that the amount of test code is 2-4x the amount of production code, TDD increases the cost of change by 2-4x when change is most likely to occur: at the beginning of development.
Agreed, it increases linearly. But now think about the cost of change without tests. The longer you wait to introduce the change the more you will have to change (increases exponentially). If this code was fully tested and maybe refactored (isolated) when the need arises, the cost of change would almost go to 0 because you would only have to implement a new feature without touching the old code.
> If this code was fully tested and maybe refactored (isolated) when the need arises, the cost of change would almost go to 0 because you would only have to implement a new feature without touching the old code.
You don't even know the "change", you just assume that it will be a change that takes zero effort to implement, to satisfy your argument.
I don't really get what testing has to do with whether or not a piece of code is isolated or not? For example if you're making a change to a 4 line function with a clearly defined interface, the change is probably going to be trivial, regardless of whether the interface is defined by a set of unit tests or a short comment above the function declaration
Likelihood of change and cost of change are different. A codebase which is very large is, all other facts held equal, much harder to change. While a small, early codebase may iterate very rapidly, drag will increase. The question is whether in the expected lifetime the higher upfront cost of providing tests pays off in average velocity.
Either the test came first and the code after or the code came first and test after.
An issue can arise in fast agile test driven development environments if you develop the code first and then do not get time to do the tests. If you start with the test you will be ensured to have time to write code otherwise the software will not work.
"The muscle doesn't see what you're holding in your hands." Arnold Schwarzenegge
- The tests doesnt care how you got them in place
In many organisations it often happens that the test team implements tests and is ready to test a feature before the dev team has actually implemented it.
IMO, that's the basis for TDD, really. When you start coding you should know what the result should be and thus you should write tests first to capture this and guide implementation.
I'm generally inclined to agree, test coverage is good, but test driven development is awkward. I think there's a tinkerer's mindset which is very much stifled by TDD.
I think TDD does shine for some things, it especially shines for refactors (where the tests ideally shouldn't change anyway) or bug-fixing where you can write in a test to catch the bug that has been found as a regression test before starting to fix the bug.
For brand new features it is a signifincant discomfort. I concede that that discomfort can result in more thought-through code but it can also just put off the implementer into not working as efficiently as they would be able to in a 'code first, then refactor' mindset.
On the flip side, "code first, then refactor" is only safe if you have tests to confirm the validity of the refactor, and in my experience, a TDD mindset is the only thing that ensures I have thorough tests by the time I need to refactor.
That said, it sounds like my version of TDD may be different than a lot of others. The tests I write before coding tend to test real-world logic and requirements, in terms of the domain language, rather than testing implementation. This means I can refactor everything behind that DSL-like facade and my tests give me assurance that I haven't changed the logic.
If your tests are at the implementation level, I can see how a TDD approach would impede refactoring and eliminate tinkering. If refactoring the code means rewriting the tests, then your tests aren't protecting you, they're just getting in the way.
Anything you write as an engineer has to be tested. Manual testing is way slower than automated testing.
I find that when I have a few good automated tests in place, especially in browser selenium tests, I save literally days of work of manual testing. Thats time that I can use to build more features, get more feedback on my product or do other more productive things than manual testing.
10 minutes of time writing a good test saves me days of work in the future testing new changes. Writing good tests is a high impact, time saving investment.
Manual testing is way slower than automated testing.
It’s entirely a matter of perspective. I consider writing a unit test to do something the compiler could check for you if you were using a statically-typed language to be “manual testing”.
I feel this is an unintentional false dichotomy caused by the name "test-driven". An alternative might be automatic-verification-driven or something like it.
Basically I want the computer to automatically point out errors, and I want to prove that it will do so by prompting that error detection first. Whether that's by types or test cases is secondary.
Personally I prefer types to test cases for many purposes (a good type system sweeps away a lot of toil in my experience), but I will use whatever tools a given ecosystem gives me in the mix that makes sense for that ecosystem.
> A great idea hiding behind a terrible implementation
TDD is a method.
TDD forces you be clear about what the result of your code should be before you start coding. It also ensures that there is a test to check your code. This is certainly not dumb.
You will write tests anyway (right?) so it does seem smarter and more effective to write them first in the same way as you write requirements first before writing code. Nothing prevents you from iterating as progress and discover corner cases, for example.
People who say that TDD moves slower are simply admitting that they don't test. Automated tests are also crucial because 99% of testing is regression testing and that obviously cannot be done manually every time the code changes.
I found it rather ironic that the article ends by praising behaviour-driven development, which is an extension of TDD...
> TDD forces you be clear about what the result of your code should be before you start coding
... which is often difficult and inefficient. A large part of the point of the article is that when making changes to an existing system, you often DON'T know until doing some experimentation what the result of your particular new bit of code should be. You know what change you want to happen to the behaviour of the system as a whole, but don't yet understand how the pieces that make up the system slot together. That permits you to dive in and try making changes and see what their effect is, but doesn't permit you to write a unit test right away.
> it is smarter and more effective to write them first.
But... why? You're asserting this without any justification.
> People who say that TDD moves slower are simply admitting that they don't test.
This accusation is pretty tedious. The article was explicitly comparing the test-first-code-second approach against the code-first-test-second approach and giving a reasoned argument (which you've not acknowledged or engaged with) for why the second was usually superior. The author isn't saying they don't test or arguing for not testing.
> I found it rather ironic that the article ends by praising behaviour-driven development
If you start coding before knowing what you are implementing then you have a bigger problem than TDD.
Experimentation is not the same as implementation, by the way.
Obviously nothing is adhered to 100% of the time in real life, this is not a dogma, but the point that you should know what the result of your code should be before you write it and should thus write tests first (which can also help you clarify your understanding) is eminently reasonable and valid.
> If a coding style turns out to be great in theory but not practice, let’s keep the parts that work and toss the rest. My convoluted point is: There are lots of ways to code, we need to stop holding any one of them up as the “best” way.
I think this is the crux of what the authour is trying to say. Not necessarily a criticism of TDD, but rather a criticism of holding to a particular belief no matter the circumstances. Of being inflexible.
I don't disagree with being flexible when needed but the counterargument is that flexibility can sometimes be used as an argument for corner cutting. For example when you start to make a few exceptions for TDD it then becomes easier to make more exceptions; if you're not careful you can quickly end up in a situation where your rational is "we don't have time to write tests" or worse yet "can't be bothered to". The point of methodologies like TDD is they force people to follow the best practices even when it's easier not to simply by mandating a standard process.
We start with hard "every module must have an end-to-end test" generally. This is a good level. Add tests as needed for debugging issues.
Lately I've hit some joker boss who thinks TDD can be done without automated tests. :(
The tests are semi manual (scripted), relatively slow to execute and a development bottleneck as the test machines are few. Some other jokers devs ignored even this limited testing and obvious breakage results.
> The point of methodologies like TDD is they force people to follow the best practices even when it's easier not to simply by mandating a standard process.
I don't disagree with that, and I don't think the authour disagrees with that.
It's when the mandated process interferes with working in the actual world.
A better test-pipeline in the article's example might be, "We don't merge without full test coverage of changes" rather than "We must have a test before we've explored how a spec should look".
If the standard makes the real world overly difficult, then it will cause people to try and cut corners. It makes the developer seek to do the wrong thing.
There's nothing inherently wrong with TDD or a bunch of other processes. There is a problem if you're pushing yourself into a corner where you find yourself fighting the process rather than solving the task that needs solving.
> A better test-pipeline in the article's example might be, "We don't merge without full test coverage of changes" rather than "We must have a test before we've explored how a spec should look".
This is very much the rule I prefer to enforce as well. Though I do really like the TDD approach when there are defined specifications or refactoring happens. In those situations it can sometimes even speed up development because once the tests are in place you then have a tighter "write, test, repeat" cycle.
However I do still agree with all the points you've raised.
I have learned and used TDD before, but I don't do TDD. I am, however glad I learned it, as it influences the way I write code now to the better. (Think more before I start writing, for instance, and writing smaller testable units of code up front). Much in the same way learning a new programming language/paradigm also can change how you write code in the old language.
Regardless of my opinion on the topic, which I don't have:
why must software engineering be mainly opinion-based?
Why can't we have objective facts, or at least only talk about objective facts without going out asserting one's opinions as truth?
AFAIK other engineering disciplines do not suffer from this problem as much as software engineering. This really bugs me.
Well call me crazy, but imho the reason for that is that for the most part, software engineering is not engineering. We're not fighting laws of nature, we're fighting other people's style and decisions. That's why our work is opinion based: we're basically computer lawyers.
Software engineering may not be an engineering discipline but in theory you could set up empirically studies/experiments to test certain hypotheses of software development practices. In theory you could also review or evaluate design against certain criteria and derive what's better to fulfill certain goals.
And yet, when the subject boils down to people and culture, testing is slow, complicated and unrewarding. Results need a lot of variable elimination and are rarely used for practical decision making. This applies to most “people” subjects.
Testing nature is much easier, and thus establishing norms in these subjects is easier.
I can't agree with this. We are fighting with chaos, we organize and encode processes, systems and information.
This is the definition of engineering on wikipedia and we fit the description:
> Engineering is the creative application of science, mathematical methods, and empirical evidence to the innovation, design, construction, operation and maintenance of structures, machines, materials, devices, systems, processes, and organizations.
That's because there are too many developers with the title of senior but skills of junior, and a business person can't tell which one is which unless they're one of the few high experienced developers out there.
The only time where a business person knows they hired the wrong people is when people claim for a rewrite. At that point, sorry, it's too late.
Because this is really project management not software engineering. You would get into real arguments about how to micro manage a team of engineers designing a space shuttle replacement. Complex expensive tasks with real tradeoffs don’t have simple answers.
There is a good amount of research on software engineering, I love love love "Making Software: What Really Works, and Why We Believe It" by Andy Oram, as it demonstrates the non-opinion-based way of thinking about software engineering. And has some really interesting insights to boot.
Because software projects fail for people reasons, and people reasons are usually subjective.
If you have a tight team of experienced devs building a simple CRUD app you can do almost anything short of building the thing in Polish BASIC with triple spaced lines, and you'll still get a result. Likewise if you have a team with 8 managers and your best dev has only done a couple of courses on codecademy, no set of engineering approaches is going to save you.
And that's just the factors within the engineering team. For every inexperienced team that doesn't deliver a result, there's 10 teams that are gutted from internal politics or management decisions, and 100 startups that fail to find a market fit. On the flip side of that, every month or so I find either a showstopping bug or a new puzzler of a UX decision in one of Uber's products, but it literally has no bearing on their success as a company. The simple fact is it doesn't matter whether they do TDD, or scrum, or whatever, as long as they don't do anything truly catastrophic.
Basically, the majority of success factors for the majority of projects come from outside the engineering team. And the ones that don't will be dominated by the team's basic development skills, which makes the signal much worse for measuring the effectiveness of anything from a project management, high level architecture, or testing perspective.
As I have more and more experience in the software development field, the more I think it is not engineering, but rather art. At least it's a creative process, not always an exact science.
I think the question is easier to answer if it's flipped. Why can't other engineering disciples work fast-and-loose?
The answer is that for software engineering, the cost balance between "working fast" and "avoiding errors" largely works out in favour of "working fast", even with the potential error rates factored in. In other engineering disciplines, the cost of failure is higher (often much higher), and therefore "slow and steady, avoiding errors" methodologies win out in this balance.
First, software "engineering" is not an engineering discipline. You might as well call novel writing "literary engineering", but calling it that won't make it true.
Second, the 10x software developer is very much real. People try to ascribe their performance to this or that technology or methodology, but in reality what makes a 10x engineer is a special psychological predilection, not their tools.
It seems to me that software tests is like checking if an architectural design is structurally stable.
It's dumb to do that when you still don't have the characteristics of the space where you are building or the materials available, but it's dumb not to do it when you know exactly what and how you are building it.
In the same way, it's dumb not to do it when you have already a building and you are going to start to change things.
I think the problem is that, being all information, in programing those phases look the same but they are actually very different.
In my experience, if you find that you're unable to do TDD with a problem you're trying to solve, then you've probably not decomposed the problem enough. In your mind. Step back. Think.
The rush to produce code, now, is making working in this industry suck.
I find that writing _something_ can help me to get a feel for a problem. Start, then sitting back and thinking about it, helps me to move towards a better solution faster.
Of course, I don't always do the stopping and thinking part...
> I find that writing _something_ can help me to get a feel for a problem. Start, then sitting back and thinking about it, helps me to move towards a better solution faster.
That summarizes the struggle that I have with TDD. I guess you could argue that I’m doing TDD wrong, but most of the time I feel like I don’t know what I’m going to be writing until I write it.
Basically most of my coding I feel like I’m prototyping until I iterate a few times and arrive at a solution I’m happy with and understand where the boundaries between various functions need to be. Until then, how am I supposed to pre-empt my solution with tests?
What I keep running across, in different companies, different domains, different engineering styles is code that was written first with the intent to add code later... that ends up being untestable. It seems to be a common “design pattern” (in Java, at least) to add a Singleton that connects to a data store and reference that Singleton everywhere in the codebase - which means that using any of the code triggers a static method that tries to connect to remote Mongo/Cassandra/Postgres/Redis/something.
However, there's a myriad of other problems TDD doesn't help you, like when you need to design a system beforehand or you want to make an educated jump to a solution that's good enough and you already know it.
I am more a fan of behaviour driven where the behaviours are written using specflow or cucumber almost like acceptance tests. Then the developer writes code but the behaviours are validated by the business analyst or product owner
175 comments
[ 3.0 ms ] story [ 257 ms ] threadAlternatively incognito or no js should do the job also.
While my dev approach is more about think, think, think harder, code, compiler barely complains, fix small issues, write test, done.
TDD felt very uncomfortable because the thinking part didn't seem to be emphasized much. It's like keep going, you iterate again real soon in this code, now again, and again.
Maybe I got it wrong, I don't know. But I share the author's frustration.
Don't seem fair to call out author on not writing tests because they never said that.
I have never actually seen somebody writing all of the tests before implementing the code even if they claimed doing TDD. Parts of tests maybe but I think that things rarely go the way the definition is spelled out.
> Don't seem fair to call out author on not writing tests because they never said that.
Fair point. I have written my comment halfway into the article, the author makes a point about tests at the end.
I had an architect who did exactly this. It was awesome, one could just look at his code and see where it would fail because it didn't occur to him to either test for that case or actually think what the code does. His only explanation would be "but it passes the tests". Later he was made into CTO.
In a way, not testing for edge cases that are found during development is quite similar to commenting on article one hasn't fully read.
This wiki has great content about the different types of TDD. Reading it helped me get useful insights about my approach and my gripes with TDD. In particular it's interesting how different approaches to testing influence heavily the design of your code.
There are so many attributes.
I think one of the biggest values of TDD is it teaches developers how to write unit testable code, which, if you are new to unit testing, can be quite hard to get your head around.
As for its usefulness in ‘real-life’, I’m going to answer with a solid ‘it depends’.
Anyone who dismisses it is missing out on some potential big advantages, anyone who is a 100% TDD zealot is probably either wasting time or lacks extensive real world experiance.
25 minutes, 16 points - all that it takes?
Regurgitating others' opinions and making multiple unfounded assertive statements is key to demonstrating he has already mastered the software domain.
It creates an unfriendly culture if people are told they can't talk about their frustrations and challenges without having x years of experience first.
But some tools require expertise and a beginner might not get the entire picture yet. E.g, if someone with 1 year of physics experience claims Quantum Mechanics to be dumb, is that a valid opinion?
If it doesn't in fact do that for this person it's totally valid to mention that.
But I used VIM for a decade and Emacs for 6 months.
Is anyone going to take my opinion on Emacs seriously, knowing that these tools take time to master?
Maybe not, but you're welcome to publish an essay on what you think.
In certain situations they may be even more valid. But when it comes to software processes you really need to have lived and breathed them in order to mount a credible argument.
>It creates an unfriendly culture if people are told they can't talk about their frustrations and challenges without having x years of experience first.
Nobody is denying this person the right to talk about their frustrations.
bjornsing's comment is implicitly highly critical of them doing so.
Something like "I dont like TDD" or "I struggle with TDD; here are my thoughts" would at least ground us in our expectations.
What does this even mean? Their experiences are valid?
He talked to a bunch of experienced devs and summarised their opinions. And then he shared his own opinion on the matter.
Nothing wrong with that.
The context of that opinion is highly relevant and not qualified in the article or a bio.
His insta page listed in the bio however gives an insight into his experience.
So yes, I feel perfectly content to raise his lack of experience.
I'm fed up of noise posing as insight.
TDD is awesome when fixing bugs (as opposed to writing features), when writing straight up business logic (especially when there are a lot of edge cases) and when you know exactly how something is going to work.
It's really bad when doing anything that requires exploration and research (which is the example in the article).
High level quality/performance check, rather than testing functions. Produces values, not pass fail. Not really a unit test, an automated integration test of a module.
Simple test run to detect crashes or utter failures in CI fashion helps with speed of development too. Also not a unit test.
Once we had a research solution and were setting up performance improvements, we have put regression tests in place for the replaced functions and the whole system, derived from earlier module quality tests. Added a few well chosen property based unit tests.
This has caught edge cases in the fast implementations.
One thing I've been trying out recently is adding asserts to code (design by contract): encode function preconditions, postconditions, invariants.
Now there are many more ways to "crash" the system while testing. The gist of it is: if you know that there is a bug if a particular condition happens, add an assert encoding this. Now all code that exercises the code with the assert automatically become tests.
I've found preconditions to be more useful than the other properties.
(Edit: pre.post/assertions/invars etc are what I've used in my own code. There's little push in most places I've worked for any testing, never mind anything systematic like this).
For anyone reading this, I originally was taught assertions etc. at university but really got to use them after reading this ppt presentation by Tony Hoare (PDF here. Only took 1/2 hour to find, thanks for getting crappier, google).
Really recommended! http://www.cs.cornell.edu/courses/cs2110/2014sp/assignments/...
Honestly, it's short, please everyone give this 10 minutes of your time. It was one of the main things that started my move from rando programmer to something closer to engineering practice. I just wish companies I worked for gave a damn for it.
IMO TDD works if your input and output is clear from the start and the way to get the output is convoluted enough that you need the test.2
It’s hard to use the data, because we’re comparing different projects, teams and suppliers but our data shows no advantage in choosing the companies that are very test-focused.
They are often slower, more expensive but have the same amount of incident reports as the companies which tests less or doesn’t do automated test at all.
That's not what you describe.
Companies that sell themselves on TDD (among other things) are part of our dataset. Maybe “automated tests” isn’t the exact literal description of that, but I didn’t mean it as such.
My conjecture: I am not so surprised at these having the same % of failures - after all the TDD will not magically lead everyone to think of all the possible ramifications; you write tests for the case that you can "imagine" just like a non-TDD developer will implement logic to cover cases they can "imagine". But on the other hand, if after fixing the bug you need to prove that you haven't caused some other side effect, then having a full test suite in place should be more efficient than retest everything again (or less risky than "ok, fixing module W should not impact X, Y and Z, so we test X only and ship...").
Also @moksly's point seems at odds with an apparently highly regarded piece of software, SQLite.
A software such as sqlite is in the rare situation that stable releases are more important than anything else.
Everyone I've worked with got payed for features though, not stability
Yet it's obvious to me that correctness and stability should be features, and more fundamental than all others. It's not as if a non-extreme amounts of testing is all that expensive.
No. The only thing that consistently affects speed is size of the suppliers, the bigger the slower.
I also think testing compensates for communication inefficiencies when developing with a larger team with developers with less tenure and experience. I'm not sure this tax is avoidable as you try and scale up organizationally.
Code architecture also has a significant effect on the cost of writing tests, and TDD was originally propounded as a way to ensure that code is testable, but with object-orientation it tends instead to lead towards dependency injection and pluggable interfaces purely for testing rather than reconfiguration, and you end up with overly abstract software.
More functional code, and architecture which increasingly represents control flow as data flow, is a better direction to aim in, IMO. That argues towards streaming and messaging techniques more than interfaces and APIs.
TDD is great for fixing bugs, less great for software design. I think the key reason is that the best design is in tension; it's a solution being applied to several problems, and triangulating that point starting from a single problem is both (a) further away from the solution, and (b) unlikely to be close to the final design unless you were really lucking in choosing the test problem.
I like this approach and it was used heavily in a previous role by both the devops and development teams in a company I worked for.
However it can come with a cost if the tests are not targeted enough or there are simply too many tests to run within a reasonable period. At one stage tests were taking 20-30 minutes to run, so you could be waiting an hour or more sometimes to find out whether the change you pushed went green or red. Often this meant people pushing before their lunchbreak or at the end of the day and effectively leaving the build broken for many hours, with others unable to push unless they included a fix for someone else's breakage.
Personally I need tests to run in a few seconds. I work on a large project that does it so it's definitely possible.
Far more holistic than the 20-30 minutes I described in that previous role, where the whole set of tests was run in a single vagrant VM, usually recycled to save time on spinning up a new VM for each run.
There's a lot of great stuff you can learn from devops, but there also seems to be a large amount of cargo culting where people have cut & pasted certain solutions from a blog or something without understanding how to adapt it to their environment and make it work properly for them.
So the type of customer you are attracts those type of devs.
Edit: this doesn't seem downvote worthy....
I'd completely believe that the automated testing companies were slower and more expensive, but I'd be inclined to blame the quality of the tests themselves. I've seen so many code bases where the tests slow down development and rarely catch errors, where you'll often have to spend more time decoding what the tests were supposed to be testing than the actual development. Even otherwise good developers can see testing as a necessary compliance item and will write awful tests just to get coverage up.
In your dataset are there any standout companies with low incident reports which do/don't have automated tests?
The more useful thing to notice beyond TDD is how this is true of almost every development model and programming pattern (including the patterns built into languages that pretend they are not) - development processes and programming patterns are almost never generally applicable - The problems come when people who don't understand this attempt to apply them generally and blindly, e.g the classic go a read a patterns book or development strategy book and now everything is a nail for their preferred choice of hammer.
I'm not being defensive of TDD though, just because it's subjectively useful does not make it equal, to me TDD looks to have very narrow utility in general for the vast majority of projects it's going to get in the way.
Therefore, we should have MDD - miniplan-driven-development: make a mini plan for the feature being developed, not so dissimilarly from what you would do with the first pass of TDD. The difference is, once you start implementing that plan and realize the problem is much different, you don't have to go back and keep modifying your original miniplan. You just do what you would normally do.
What you gain is probably a partial improvement in your development process without much cost.
DHH wrote some about it here: https://dhh.dk/2014/test-induced-design-damage.html
I do like automated tests (unit and integration) and often TDD is appropriate. However tests are not the end goal of an architecture. The code should be maintainable, easy to follow and simple to reduce number of bugs. If TDD starts to change that by adding layers of indirection or changing the purpose of something just so you can do a fast unit test then something is wrong.
I wouldn't say that, I have an counterexample ([1]): It is clear what the refactoring (that is to be implemented) should do and how to state that ([2]), but it usually requires exploration and research how to work with the AST and which methods to use.
I would even say that TDD is awesome for exploring, researching, for getting a feeling of the problem's context, what to do and what the caveats are, as long as neither of that is required for stating the test itself.
Having tests first, you can simply start the debugger, try things out and with hot reloading you can write the code while running the specific test that you want to concentrate on. Visualization tools like [3] can greatly help exploring how to solve and implement the current test while you debug it.
[1] https://blog.hediet.de/post/implementing_typescript_refactor... [2] https://github.com/hediet/hediet-ts-refactoring-lsp/blob/mas... [3] https://marketplace.visualstudio.com/items?itemName=hediet.d...
When it comes to complex systems, writing unit tests too early in the project is a bad idea; especially when combined with an agile development methodology. It's like building a skyscraper over several years and perfectly analysing every steel rod, every brick and every pane of glass as it is added to the structure but never actually stepping back to check that the structure as a whole is actually pointing upwards.
But I think we can also define it more inclusive and still be fine: TDD is a good paradigm for programming against well defined APIs.
For example if you are handed an API specification then locking down your interface(s) with tests in advance can be very helpful.
What is even more helpful is to have a higher level language or abstraction that encodes the specification in detail and generates the tests for you (Clojure spec comes to mind).
edit: typo
Your previous-to-last sentence applies equally to a software project with or without tests.
You don't even know the "change", you just assume that it will be a change that takes zero effort to implement, to satisfy your argument.
What came first the chicken or the egg?
Either the test came first and the code after or the code came first and test after.
An issue can arise in fast agile test driven development environments if you develop the code first and then do not get time to do the tests. If you start with the test you will be ensured to have time to write code otherwise the software will not work.
"The muscle doesn't see what you're holding in your hands." Arnold Schwarzenegge - The tests doesnt care how you got them in place
In many organisations it often happens that the test team implements tests and is ready to test a feature before the dev team has actually implemented it.
IMO, that's the basis for TDD, really. When you start coding you should know what the result should be and thus you should write tests first to capture this and guide implementation.
I think TDD does shine for some things, it especially shines for refactors (where the tests ideally shouldn't change anyway) or bug-fixing where you can write in a test to catch the bug that has been found as a regression test before starting to fix the bug.
For brand new features it is a signifincant discomfort. I concede that that discomfort can result in more thought-through code but it can also just put off the implementer into not working as efficiently as they would be able to in a 'code first, then refactor' mindset.
That said, it sounds like my version of TDD may be different than a lot of others. The tests I write before coding tend to test real-world logic and requirements, in terms of the domain language, rather than testing implementation. This means I can refactor everything behind that DSL-like facade and my tests give me assurance that I haven't changed the logic.
If your tests are at the implementation level, I can see how a TDD approach would impede refactoring and eliminate tinkering. If refactoring the code means rewriting the tests, then your tests aren't protecting you, they're just getting in the way.
Just having an understanding of TDD and holding it as an aim will ultimately result in a better, more maintainable codebase.
So what's the downside?
Anything you write as an engineer has to be tested. Manual testing is way slower than automated testing.
I find that when I have a few good automated tests in place, especially in browser selenium tests, I save literally days of work of manual testing. Thats time that I can use to build more features, get more feedback on my product or do other more productive things than manual testing.
10 minutes of time writing a good test saves me days of work in the future testing new changes. Writing good tests is a high impact, time saving investment.
It’s entirely a matter of perspective. I consider writing a unit test to do something the compiler could check for you if you were using a statically-typed language to be “manual testing”.
Pointless unit testing is a separate thing.
Basically I want the computer to automatically point out errors, and I want to prove that it will do so by prompting that error detection first. Whether that's by types or test cases is secondary.
Personally I prefer types to test cases for many purposes (a good type system sweeps away a lot of toil in my experience), but I will use whatever tools a given ecosystem gives me in the mix that makes sense for that ecosystem.
TDD is a method.
TDD forces you be clear about what the result of your code should be before you start coding. It also ensures that there is a test to check your code. This is certainly not dumb.
You will write tests anyway (right?) so it does seem smarter and more effective to write them first in the same way as you write requirements first before writing code. Nothing prevents you from iterating as progress and discover corner cases, for example.
People who say that TDD moves slower are simply admitting that they don't test. Automated tests are also crucial because 99% of testing is regression testing and that obviously cannot be done manually every time the code changes.
I found it rather ironic that the article ends by praising behaviour-driven development, which is an extension of TDD...
... which is often difficult and inefficient. A large part of the point of the article is that when making changes to an existing system, you often DON'T know until doing some experimentation what the result of your particular new bit of code should be. You know what change you want to happen to the behaviour of the system as a whole, but don't yet understand how the pieces that make up the system slot together. That permits you to dive in and try making changes and see what their effect is, but doesn't permit you to write a unit test right away.
> it is smarter and more effective to write them first.
But... why? You're asserting this without any justification.
> People who say that TDD moves slower are simply admitting that they don't test.
This accusation is pretty tedious. The article was explicitly comparing the test-first-code-second approach against the code-first-test-second approach and giving a reasoned argument (which you've not acknowledged or engaged with) for why the second was usually superior. The author isn't saying they don't test or arguing for not testing.
> I found it rather ironic that the article ends by praising behaviour-driven development
That was sarcasm.
Experimentation is not the same as implementation, by the way.
Obviously nothing is adhered to 100% of the time in real life, this is not a dogma, but the point that you should know what the result of your code should be before you write it and should thus write tests first (which can also help you clarify your understanding) is eminently reasonable and valid.
> That was sarcasm.
Article is poor overall. Sarcasm does not help.
I think this is the crux of what the authour is trying to say. Not necessarily a criticism of TDD, but rather a criticism of holding to a particular belief no matter the circumstances. Of being inflexible.
Lately I've hit some joker boss who thinks TDD can be done without automated tests. :(
The tests are semi manual (scripted), relatively slow to execute and a development bottleneck as the test machines are few. Some other jokers devs ignored even this limited testing and obvious breakage results.
I don't disagree with that, and I don't think the authour disagrees with that.
It's when the mandated process interferes with working in the actual world.
A better test-pipeline in the article's example might be, "We don't merge without full test coverage of changes" rather than "We must have a test before we've explored how a spec should look".
If the standard makes the real world overly difficult, then it will cause people to try and cut corners. It makes the developer seek to do the wrong thing.
There's nothing inherently wrong with TDD or a bunch of other processes. There is a problem if you're pushing yourself into a corner where you find yourself fighting the process rather than solving the task that needs solving.
This is very much the rule I prefer to enforce as well. Though I do really like the TDD approach when there are defined specifications or refactoring happens. In those situations it can sometimes even speed up development because once the tests are in place you then have a tighter "write, test, repeat" cycle.
However I do still agree with all the points you've raised.
why must software engineering be mainly opinion-based? Why can't we have objective facts, or at least only talk about objective facts without going out asserting one's opinions as truth?
AFAIK other engineering disciplines do not suffer from this problem as much as software engineering. This really bugs me.
I like this. I mean, as a description, not so much as a fact, but it is what it is :D
Testing nature is much easier, and thus establishing norms in these subjects is easier.
This is the definition of engineering on wikipedia and we fit the description:
> Engineering is the creative application of science, mathematical methods, and empirical evidence to the innovation, design, construction, operation and maintenance of structures, machines, materials, devices, systems, processes, and organizations.
The only time where a business person knows they hired the wrong people is when people claim for a rewrite. At that point, sorry, it's too late.
If you have a tight team of experienced devs building a simple CRUD app you can do almost anything short of building the thing in Polish BASIC with triple spaced lines, and you'll still get a result. Likewise if you have a team with 8 managers and your best dev has only done a couple of courses on codecademy, no set of engineering approaches is going to save you.
And that's just the factors within the engineering team. For every inexperienced team that doesn't deliver a result, there's 10 teams that are gutted from internal politics or management decisions, and 100 startups that fail to find a market fit. On the flip side of that, every month or so I find either a showstopping bug or a new puzzler of a UX decision in one of Uber's products, but it literally has no bearing on their success as a company. The simple fact is it doesn't matter whether they do TDD, or scrum, or whatever, as long as they don't do anything truly catastrophic.
Basically, the majority of success factors for the majority of projects come from outside the engineering team. And the ones that don't will be dominated by the team's basic development skills, which makes the signal much worse for measuring the effectiveness of anything from a project management, high level architecture, or testing perspective.
The answer is that for software engineering, the cost balance between "working fast" and "avoiding errors" largely works out in favour of "working fast", even with the potential error rates factored in. In other engineering disciplines, the cost of failure is higher (often much higher), and therefore "slow and steady, avoiding errors" methodologies win out in this balance.
First, software "engineering" is not an engineering discipline. You might as well call novel writing "literary engineering", but calling it that won't make it true.
Second, the 10x software developer is very much real. People try to ascribe their performance to this or that technology or methodology, but in reality what makes a 10x engineer is a special psychological predilection, not their tools.
It's dumb to do that when you still don't have the characteristics of the space where you are building or the materials available, but it's dumb not to do it when you know exactly what and how you are building it.
In the same way, it's dumb not to do it when you have already a building and you are going to start to change things.
I think the problem is that, being all information, in programing those phases look the same but they are actually very different.
The rush to produce code, now, is making working in this industry suck.
Who wants to think, when you can code, right?
Of course, I don't always do the stopping and thinking part...
That summarizes the struggle that I have with TDD. I guess you could argue that I’m doing TDD wrong, but most of the time I feel like I don’t know what I’m going to be writing until I write it.
Basically most of my coding I feel like I’m prototyping until I iterate a few times and arrive at a solution I’m happy with and understand where the boundaries between various functions need to be. Until then, how am I supposed to pre-empt my solution with tests?
IMHO only thing dumb is developing without testing
However, there's a myriad of other problems TDD doesn't help you, like when you need to design a system beforehand or you want to make an educated jump to a solution that's good enough and you already know it.