This is a great post and I agree with almost everything in it 100% except for this part:
> And—yes—you can read this as my being unlikely to use Rails the next time I'm building a traditional multi-page application.
Really? The way I do it is I write a client library for the API and run the tests in Rspec on top of the client library. Sure I'm going through the whole stack, but the thing operates as fast as you can serve normal users! Surely your large project can handle 5000 concurrent requests? While that might seem like a lot of buildout (scalable testing servers), I'd argue the opposite is a lot of buildout: writing an application without Rails.
Granted my approach might be most feasible for Rails API + Ember, but I really don't see the speed of my tests passing as what's holding me back.
The other thing that is nice is that once you start needing to speed up endpoints by pulling them into a compiled language, you can keep your tests! Everything is going through the API anyway! So feel free to Nim or Rust the CPU heavy endpoint or Redis-cache the write-once-in-a-while-and-keep endpoint.
It will hurt when your end-to-end tests (minus UI) must be run per check-in and as your code gets bigger.
Some companies invest in huge infrastructure to speed things up (paralleling until you can't...) but in that situation, suddenly Devs can no longer build locally without biting their tongue.
Setting up end-to-end test "per test-case" is also tedious (should user be logged-in in order for this test-case to run? what should be the state before the tests are run and what should be the state after the tests are run? what if the state of the system have the expected _and_ more e.g.: side-effect?)
> Should user be logged-in in order for this test-case to run?
Why not? This is a simple helper function or easily parallelizable block. Same with creating a sub-resource before modifying it. People act like this is the end of the world, but it isn't. And if you like, you can still do your (in my controversial opinion) silly unit tests and you can still parallelize those too!
Whether a dev can build a project locally or not does not relate to whether or not the tests can be run in parallel.
Oh, for sure, it's a simple helper function but the ramification is huge!
1. Which user? which role should the test written as?
2. Suddenly bug showed up because expectation between roles differ (usually the bug is in the test framework)
No, it's not the end of the world, just programmer's productivity go down...people can still work for the project just isn't fast as it used to be anymore...
Compared to unit tests where the concept of the user doesn't even exist or needs to be shimmed in with a proxy object, I really don't think the user differences in ACL are the deciding factor between the two.
As for programmer productivity, proper parallelization of these tests results in sub second test suite runtime. Maybe more if there is a new dependency that the testing servers need to install, but it really isn't that bad.
>It will hurt when your end-to-end tests (minus UI) must be run per check-in and as your code gets bigger.
>
>Some companies invest in huge infrastructure to speed things up (paralleling until you can't...) but in that situation, suddenly Devs can no longer build locally without biting their tongue.
That's why you pick a few smoke tests to run before checking the code in and let the full regression test suite run while you're doing other stuff.
>Setting up end-to-end test "per test-case" is also tedious (should user be logged-in in order for this test-case to run? what should be the state before the tests are run and what should be the state after the tests are run? what if the state of the system have the expected _and_ more e.g.: side-effect?)
You need a decent integration testing framework to help here.
After you get done writing a section of code, take a few minutes to look at it and see if there is any way it can be improved. Move functionality around, rename some variables. Do it right away, while it's still fresh in your mind.
I find the best time to do this is when I come back to code that I wrote a few months ago. If it doesn't make sense immediately, then it probably needs refactored (I wrote the code I ought to understand it quickly).
Ideally it should be refactored right away as you say, but often it takes forgetting the code, then rereading it to see how unintuitive it was the first time.
In a similar vein, I like to take a break if possible, before a larger commit, then come back to it and see if I should refactor first.
Two benefits of this are (1) a fresh mind to see what you might have missed especially since when I'm done with a piece of functionality I often just want to commit it and move (2) it gets me back in the flow of development quickly.
I see unit tests recommended all the time, but in practice I've found them significantly less helpful than integration and acceptance tests. Maybe its just a nomenclature thing? It's obvious if you change a "unit" of code, its tests are going to fail. That's not terribly helpful while refactoring. What's more useful is to find out what else breaks and that's where the other tests come in.
I'm not saying that unit tests aren't useful. They definitely help for example if you're using a dynamic programming language or if you're practicing TDD. I would just emphasize a higher priority for integration and acceptance tests.
I try to keep my unit tests focused on the api of the unit I'm developing. If I'm refactoring and keeping the api the same, I shouldn't have to change any tests. Since I started writing my tests this way, I've felt I have to change my tests far less... Sometimes I add more tests, when a bug is found and needs fixing, but missed tests are better than wrong tests IMHO.
That's always been difficult for me if a unit test is written as purely as possible (i.e. isolating the unit from its dependencies). If I have to mock dependencies, create stubs and spies, I start to have to make assumptions about how the unit works beyond just its API, which makes the test itself more brittle. How do you approach this situation?
All the dependencies of a unit are part of its API. You may not need to give them at every call, but you do need to give them atleast once. Whether the unit calls MissileLauncher::launch or not is in my opinion part of the external behavior of the unit.
I feel like I shouldn't be testing that the dependencies' api's didn't change, that's for integration tests, which don't need to run as often as unit tests. Therefore, they can be slower and more complex. Mock or spy everything external and nothing internal.
It can feel a little dangerous, but as long as the integration and e2e tests are kept up and run on build, i think I'm covered. I've definitely been bitten before, but I've been bitten more by over complicated and long running tests.
Unit-Tests can be helpful when working with Java as they can give you a semi REPL-like environment (especially if your IDE integrates well with JUnit).
I find both Integration and Acceptance Tests could potentially hide many bad internal designs (tight coupling, etc). I also find that not many places can speed up their full integration/acceptance tests suites; suddenly build becomes way too painful, be it local dev build or CI-triggered-through-commit builds.
Note: My experience with super-slow integration tests choking build is with a large JavaScript project with 100 developers (at least).
I implemented integration-tests for a Java-heavy based project before but limited only to test the database component of the project (e.g.: Repository, JPQL, JPA), the ability to test business logics that are tightly coupled with ORM quickly in isolated manner is a blessing.
They must be testing "behavior". If you are refactoring to clean up code, you'll want to verify the behavior and unit tests gives you that quick feedback.
This response has been trotted out a few times on HN. IMHO its terribly mistaken.
An analogy is that unit tests are like a brick layer's spirit-level and plumb-line. As you build the wall, you ensure the pieces are put together correctly and head off mistakes early.
The "only test at a high-level" is like saying don't confirm the bricks you are laying are correct, instead bring in an inspector to check the whole building is up to code once we have finished.
If we do bring in the inspector she takes one look and says the wall have to come down because of faults right at the bottom or because we forgot the damp course or we never get that far because we couldn't fit the roof because all the walls bowed out.
I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases and that they are not in a position to be advising others.
The situation you are describing is best handled by generative tests (i.e. Quickcheck).
And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time.
Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with.
Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?
I'm not a Haskell programmer but how would you deal with a case like e.g. using a set of complex regexes for parsing some unusual date-time formats. I would want to unit test the hell out of them covering all the variations and exceptions I could think of.
I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
In this particular case I would use some Quickcheck-like generative test library and give it a description of how to generate random dates (i.e. start by generating a day, then a month, then a combination of both etc) and then leave to it the generation of all test data.
The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite.
On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense).
Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help.
However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...)
(I know this is a very short answer but I'm typing on a phone)
You can make a finite state machine to model your expectations on random input, but then your test code is almost as complex as the code :)
I've found quickcheck super useful as a way to fuzz user input or find edge cases in unit tests I didn't see, but I still write all my integration tests by hand because they are so hard to formalize.
What? On the codebase I work on, integration tests run every single commit, just like unit tests. They would both fail at the same time. Except the integration tests are much less brittle.
Yes, but I can run unit tests while I'm coding. Waiting for a check-in is far too late. If I've broken something, it's more helpful to let me know about it ten seconds later, rather than hours later, when I've pushed my change and broken the build.
My unit test suite runs in ~500ms. I run it every single time I save a file. In fact, it's set up with a filesystem watcher to do this automatically and announce the result to me via growl.
The faster your test suite, the more often you can run it, and the faster you get feedback on your code.
You can combine this with a healthy integration test suite, and only run the unit tests.
Validation rules is a good example, if I want to test that my user input validation rules are correct, I don't need to do this through the entire application frontend - I can just check those rules in isolation.
>The faster your test suite, the more often you can run it, and the faster you get feedback on your code.
In all likelihood, the faster it is, the more tightly coupled it will be. You're simply exchanging computing power for brain power (refactoring tests when you refactor code), which is nearly always a bad trade off.
Usually my unit tests are tightly coupled to the module they're testing, but the modules themselves are loosely coupled from the rest of the system.
In most cases I want to preserve existing behaviour, so I don't want any existing tests to fail.
I'll add tests for the new behaviour in the relevant modules, add code to make them pass, and then if necessary update the glue code / integration tests which cross the boundary.
Pretty much by definition, a refactor of code should not change its behaviour. My tests usually assert on behaviour observed at the boundaries of modules, so they should still pass during refactorings.
> Pretty much by definition, a refactor of code should not change its behaviour.
Quite precisely by definition, perhaps for other reasons as well :-P
> My tests usually assert on behaviour observed at the boundaries of modules, so they should still pass during refactorings.
I find the kind of refactoring that most desperately needs the support of tests is changing the boundaries of the individual units. Unit tests can still arguably be useful there in making me clarify the changes I'm making to my interfaces... but they do need changing, and I can't be quite as confident that nothing is breaking if I'm changing both code and tests in tandem.
This sounds hellish, I don't want red things flashing in my terminal while I'm in the middle of changing some function signatures; I know it's broken, I haven't gotten around to writing all the boilerplate to convince the tests I know about it yet though.
Growl doesn't flash red things in the terminal and nobody says you have to use Growl notifications if you don't like them, but this really has nothing to do with the usefulness of unit testing.
>I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases
That's funny. I suspect that people who advance the pro-unit test view do so haven't had sufficient experience with large complex code bases.
Experience with large complex code bases with a spaghetti problem is the exact thing that put me off unit tests.
Unit tests are by their nature very tightly coupled to the underlying implementation, which is why they cause so many problems.
It's kind of like pouring concrete over a bunch of code bricks: if those bricks are in the wrong place and need refactoring, by pouring unit tests over them means you're solidifying the broken architecture and making it even harder to fix.
IME there basically don't exist large projects without technical debt problems either, so arguing that unit tests work well when the architecture is good is kind of a moot point.
First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test. That doesn't help you if you are adding tests to an existing project that's a mess, though.
> Unit tests are by their nature very tightly coupled to the underlying implementation
No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. It shouldn't care about the internal behavior of the "unit". That's the theory. They're only supposed to be coupled to the interface, not to the implementation.
>First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test.
I've found this to be one of the biggest myths of unit test driven development. Developers who have a tendency to write spaghetti do so anyhow. When it becomes too hard to write tests they simply don't write tests or they write 'pretend' tests.
It's perfectly possible to write spaghetti code and write a badly written test that notionally covers it.
Good coders code well. Bad coders don't. TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation.
>That doesn't help you if you are adding tests to an existing project that's a mess, though.
Harms you deeply, in fact.
>No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior.
i.e. it couples itself to a unit, which is an underlying implementation detail.
If it's coupling itself to an externally facing interface it's an integration test or an end to end test.
>It shouldn't care about the internal behavior of the "unit".
No, but then again, since "units" are supposed to be the smallest blocks of code that comprise your code base, it's caring very much about how all of the blocks talk to one another, which is an implementation detail (for your project). Perhaps the most crucial implementation detail, in fact.
How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt.
Pouring test cement in those cracks means that any technical debt accrued in those API contracts becomes impossible to refactor without breaking the tests.
In practice this makes that refactoring more difficult than if it had no tests at all due to the massive psychological inhibition against deleting tests or turning the build red and the added problem that real failures look the same as changed API contracts.
> It's perfectly possible to write spaghetti code and write a badly written test that notionally covers it.
Absolutely.
> Good coders code well. Bad coders don't.
Also true.
> TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation.
I've used TDD and not used it. And when programming using TDD, my code was less "spaghetti" - more modular, better-designed interfaces. There definitely was causation, in my case - it made me write my non-test code differently. So that's anecdote and not data, but holding the programmer constant and changing the methodology, the style and quality of the code changed.
> >No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior.
> i.e. it couples itself to a unit, which is an underlying implementation detail.
> If it's coupling itself to an externally facing interface it's an integration test or an end to end test.
Let's get specific, so that our terms can be better defined, and we can make sure that we're not just arguing about definitions of terms. Instead of "unit", let's say "C++ class". Now: By my definitions, my unit test should test the behavior of the public API of that class, not the private implementation. So the API is an implementation detail of the project, but not of the class. I will agree that the unit test definitely couples to the interface of the class. Done right, though, it doesn't couple to the implementation of the class.
> How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt.
A bigger source of technical debt than the bugs you can't find that are buried in the implementation of some class? I doubt it. And good tests mean that those bugs don't make it very far before getting exposed.
A bigger source of technical debt than that API that you don't know how to use, and so you call it wrong? Tests can actually serve as documentation for how to use an API. You have non-test documentation? You're lucky to have that, but the problem is, when the code implementing the class changes, the documentation might well not. If that happens with the tests, you find out the first time you run them.
>Let's get specific, so that our terms can be better defined, and we can make sure that we're not just arguing about definitions of terms. Instead of "unit", let's say "C++ class". Now: By my definitions, my unit test should test the behavior of the public API of that class, not the private implementation. So the API is an implementation detail of the project, but not of the class. I will agree that the unit test definitely couples to the interface of the class. Done right, though, it doesn't couple to the implementation of the class.
In almost all cases (the exception being if the class is the core of a library's externally facing API), the class itself is an implementation detail.
>A bigger source of technical debt than the bugs you can't find that are buried in the implementation of some class?
Yes, obviously. Bugs are not technical debt.
>And good tests mean that those bugs don't make it very far before getting exposed.
And integration tests are generally better at finding said bugs than unit tests since they test more realistically.
>A bigger source of technical debt than that API that you don't know how to use, and so you call it wrong?
Is again, a bug - not technical debt.
Unit tests are horrible for this since they require you to mock APIs that you're calling. The test which mocks the API wrongly won't catch that bug but the integration test that uses the real API will.
>Tests can actually serve as documentation for how to use an API.
Yes they can. Integration tests can double as a specification for an entire application which is even better than a documentation for a (potentially) badly architected API.
I treat each unit test as an individual assertion on the logic of the program. I find using a <methodName>_<Conditions>_Expectations syntax documents the intent very well. E.g:
testCalculateGrade_scoreOver85_returnsAnA()
You're right that refactoring the code can break tests. That's when you have to ask if the original assertion still stands. It's OK to change them if your understand of the problem has changed also.
Both, unit and integration tests, are imperative. Integration tests point in the general direction of where an issue might be but unit tests are more precise.
In my experience, if you try to do without unit tests, your integration tests will grow to fill in the gap. This will slow you down in a two-fold way:
1. As your integration tests will both more numerous and complex to cover corner case scenarios, the time it takes to execute the suite will grow too. And since the integration test is more opaque too, this price has to be paid by both you and all the developers that touch something that might have used your module.
2. The time at which you receive the feedback of the test is already too late. You are no longer doing experimental changes, you are trying to push this unit of work into the code base and be done with it. Either you will be tempted to gloss over any divergence you find (quality price) or you will have to execute some relevant subset of your integration testsuite during your coding/OODA loop (execution speed price).
I haven't found this to be a problem at all. Integration tests follow a power law. If you segregate your tests into "smoke" and "non-smoke", run the former before merging code and the latter continuously and randomly and in parallel, you'll get notification of 95% of regressions within 5 minutes and the other 5% within a few hours.
Choosing unit tests because of speed is nearly always a premature optimization.
IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode.
Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).
The main benefit of unit tests is the speed one can run them at, and the feedback. Unit tests make it vastly faster to figure out the culprit in a broken sector of code almost all the time, since it narrows down what code you have to check for any logic errors. It also so happens that unit tests tend to encourage better writing of code into reuseable units.
Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play.
A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.
I've found that where code involves complex logic, unit tests often are more effective for the reasons you gave.
However, a lot (perhaps most) code is more about integrating disparate components (browsers, lots of different kinds of external APIs, databases, times, distributed task processors, etc.) rather than lots of complex logic and for that kind of code, unit tests are substantially less effective and more horrible to use whereas integration tests really shine.
I think ideally, you should keep the two kinds of code strongly segregated, with clean APIs between them, so you can easily use the appropriate kind of testing for either one.
I don't agree that a good code base necessarily has both types. A web app with relatively shallow logic and lots of integrations only really benefits from having integration tests. A library to convert human readable strings into dates likewise gains nothing from having integration tests.
Similarly, a messy spaghetti codebase with a disgusting mix of both types of code can only really benefit from integration testing until it's substantially refactored and the two types of code are segregated.
> you should keep the two kinds of code strongly segregated, with clean APIs between them
Why _strongly_ segregated?
Nothing stops you to write unit tests for the methods that should be covered by unit tests and integration tests for the functionality that needs to be covered by integration tests.
If the code is not well segregated you won't be able to write a clean unit test that tests the complex logic without also mocking a bunch of stuff.
That test will also often break as you refactor the various connecting APIs it surrounds.
Tests that break in the presence of changed API contracts (as opposed to bugs) are vastly less valuable than tests which only go red in the presence of a bug.
I have different point of views. For me, Integration tests are always better.
I normally structure two type of integrations test:
5-10 minutes tests that run before every checking.
Over night, over weekend tests that runs every night and every weekend.
5-10 minutes are enough time for 200 UX features level tests for both embedded system and Web backend/frontend integration type project, for me,
With those test coverage in place. The worst time I receive the feedback from the test failures are one day or one weekend. All the changes are all within my mind that might cost the failure.
BTW, ALL checkin code should be passing ALL integration tests. If they don't, the highest priority is fixed the test failures and not develop any new feature.
> It's obvious if you change a "unit" of code, its tests are going to fail.
No it's not. Not at all. The whole point is you can change the guts of a unit around and know you haven't broken things cause the unit tests still pass. Because the unit test is testing the "contract" (explicit or implied) the unit has made with other units about it's inputs and outputs.
Unit test should fail (and require updating) only if the unit's INTERFACE changes.
But, if your unit tests are failing everytime you touch your code then 1) your code is not cleanly isolated and independent and/or 2) you have poor unittests.
If nothing in an application depends on some aspects of the behavior of a unit interface, what is the point in testing it?
You only need that sort of test in a programming language or public library, because you're publishing the behavior to unknown numbers of users, and it has to work, as documented. If they find too many things that don't work according to the documentation, they will correctly suspect that you aren't testing, and lose confidence.
But if some API is internal to an application, and no code depends on some aspect of its interface contract, then that aspect effectively doesn't exist.
It may happen that someone discovers this aspect in the documentation and tries using it. At that time it is either found broken, or found working. If it is found broken, it can be fixed. Or else it can be striken from the documentation (and another way can be found in the application code to meet its requirements).
If the aspect is found correct, or fixed to reflect the documentation, and is then used in the application, then that use constitutes a test. The feature is tested indirectly, through some regression test case which targets something in the application, which depends on that aspect of the API contract lower down.
If you design units with all kinds of detailed aspects in their interface contracts which end up not being used at all, and test all of them, it can be argued that you're wasting time: you're basically working on your library design hobby, at the expense of the project, which just serves as a vehicle for it.
Unit tests can verify assumptions that the implementation is internally relying on. E.g., say a greedy algorithm relies on a collection being sorted, even though the public interface does not return results in any particular order. If I make a mistake such that an edge case leaves the collection not sorted correctly, a test of the public API can't tell unless it happens to trigger the edge case whose importance might not be obvious.
I think I'm arguing for letting white-box unit tests make assertions about implementation details in a simple and clear way, rather than having to somehow devise a test using only the black-box API that happens to fail if the assertion doesn't hold. In other words, you can only guarantee parts of the internal state are valid if you know how to infer them from outside.
Are you arguing that all unit tests should be black-box?
Refactoring is defined as the process by which you restructure code without changing its external behaviour. So in other words, refactoring should never break tests because the external features of the code do not change.
Now - redesign or rewriting code is a very different thing. Refactoring can lay the groundwork for redesign or component level rewrites. But the process of redesign and rewrites could result in breakages unless the right level of isolating layers are put in place. This usually involves lots of work to keep tests working.
Personally I have found that unit tests don't result in less work. They are a LOT of work and will continue to be a lot of work if they are to generate value. What they offer though is the ability to know when things have gone wrong and the confidence to release and deploy. I realize that you don't get increased confidence in your release process by doing less work. Software it appears, like almost everything else, obeys the basic laws of thermodynamics. :)
I think the problem here is that I didn't specify exactly what I meant by "refactoring", "unit" and what types of changes I would be making. Often times I find refactoring necessary because a significant piece of the system was either under or over engineered and it becomes necessary to update that code without changing the functioning of the system. So by refactoring, I mean refactoring at a level the encompasses multiple units of code, while maintaining the same functioning of the system. That means the interfaces may change for the individual units, which would obviously break the unit tests.
That's a really poor article, and doesn't really speak to integration tests in the first place. I'll follow up on the article's deficiencies around the argument they were trying to make when I have a bit more time free.
As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported.
The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well.
To the weaknesses of the article with regard to its critique of end-to-end tests:
> Let's assume the team already has some fantastic test infrastructure in place. Every night:
Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests.
> To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete.
It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to:
> A failing test does not directly benefit the user.
A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied!
> End-to-end tests were flaky at times.
Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests.
Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests.
It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.
>When the tests do run, flaky tests tend to be a fact of life.
Flaky tests are indicative of bugs in your tests.
Google is literally advocating treating your bugs as a fact of life, not fixing them, and avoiding harder engineering challenges because they are harder.
They serve different roles in my projects. Unit tests are intended to A) define and ensure adherence to a contract, a role which can also be partially (though not fully) fulfilled by strong and/or static typing in some languages; and B) as a harness to drive development while it is ongoing, which is a transient function, meaning those tests can be removed or consolidated after a development cycle.
Integration and functional tests, on the other hand, are long-lived tests intended to ensure that all the pieces stay glued together. They allow the middle parts of code to squiggle quite a bit, as long as they get to the same place at the end of the line. They allow for safe refactoring of modular units whose entire scope lies beneath the boundary of the test. And when the functionality of the application or library changes, they should change as well, but this should be an order or two of magnitude less frequent than the unit test level.
As a rule of thumb, I put integration tests around any piece of functionality that "makes me money"—less literally, around any bit that, if it were broken, would prevent things like user signups, users paying the correct amount, or critical functionality that is indispensible to a user of the software. Likewise, I usually write unit tests when i'm writing new modules so that I don't have to keep reloading a browser or setting up complex data schemes while they're in development. Sometimes I remove those, especially for code that is low-churn (which is a totally under-appreciated metric to look at when you're figuring out where you should focus coverage, IMHO.)
Both types are important necessary. And not all of them are destined to live forever.
Maybe its just a nomenclature thing? It's obvious if you change a "unit" of code, its tests are going to fail.
Here's the thing about "agile" and Unit Tests. The current generation of agile was re-discovered in Smalltalk, after the Smalltalk Refactoring Browser had arrived on the scene. Not only could you refactor at warp speed and high confidence (1) and with unlimited undo, since Smalltalk SUnit test were just ordinary Smalltalk code, Unit tests often got refactored right along with the code.
If you're in an environment where you have to painstakingly refactor, then also separately update your Unit Tests, then it's going to feel like you're doing a whole lot more work -- because you are!
Given that very few environments have the requisite properties, TDD is going to feel like a slog on most stacks, and people will naturally gravitate to something lighter, like integration/acceptance/smoke tests.
(1) - Yes, the prepackaged Refactorings in the original Smalltalk RB were guaranteed to be correct. There were sophisticated runtime checks, and they'd refuse to fire off if correctness wasn't completely certain. You could fool them -- because dynamic language -- but you had to try pretty hard to mess up like that. Then the rest of the shop got really annoyed at you!
I liked the part about "technical debt".
Yet I don't like this statement: "You should aspire to upgrade your dependencies and frameworks all the time". It's too generic.
I'm thinking about third party dependencies here (eg via package managers). In my opinion it's best practice to stick with one certain stable version and then stop updating it - to avoid wasting time on possible needed refactoring due to changed dependency behaviour.
Monitoring the change log of the dependencies to see if security issues are fixed and reacting on that is of course still reasonable.
Yet upgrading just because " upgrade as often as possible" is not.
Maybe it's different for projects lasting 10years or more, maybe there are other rules to make. My experience is with 2.5yr-projects in average.
Maybe I'm working with dependencies which get updated every week or so, maybe the author's dependencies are updated every half-a-year.
If you're not tracking the changes of your dependencies, you're moving that work (finding and fixing the problems) to your future self/team. That might be a win if future you won't need to continue to support/develop the thing. I think it's a loss in other cases. IME, you end up trying to swallow a sea of changes all at once, sometimes triggered by something that you can't control. "Oh, I need to take the new rev of foo; that needs a newer version of bar and baz. Baz needs a new quux..." Suddenly, you've got a lot of fixing and regression to do, and if the need for the new foo comes at a bad time, you may have lost control over your schedule predictability entirely.
Moving an expense from today to the future is debt.
The point about the need to upgrade dependencies as being sometimes forced by events beyond our control is a good one.
Debt (technical or otherwise) in itself is not a bad thing if used sensibly for some gain. Allocating resources to maintain dependencies is not necessarily the best short or long term investment.
I think it comes down to what kind of product you're building and what kind of libraries you're using. If you're doing some web-facing product that's being continually developed, and there's a security update in a library or framework, then sure.
If you're building a traditional piece of software (no "evergreen" continual updates, maybe even with no upgrade process at all eg. device firmware) then just using a library internally, and your requirements aren't changing, then continually updating it is busywork at best, and a risk to the reliability and stability of your project at worst.
It's not "technical debt" to skip point releases that don't add anything to your project, since you avoid potentially re-integrating changes to the same features if they change multiple times, and there's a good chance you'll never need to update at all.
I've found in practice the effort to go from a -> b -> c -> d in aggregate is often far less than the effort to go from a -> d.
In addition I've found the stability of the system as a whole is better when you swallow the pie in small chunks and when your project depends on multiple dependencies upgrading them gradually piece at a time makes bisections much simpler than "stop the world and update".
It really does depend (as most things do) on the particular domain/problem/solution though, if I was working on something that ran for years and couldn't be touched in production I'd probably go the other way.
That approach might have worked 20 years ago when a big project had more than 3 3rd-party libraries and CPAN was the pinnacle of dependency management, but it's burying your head in the sand in this day and age. Security patches are a fact of life, and unless you've air-gapped everything, aren't optional. Sooner or later you're bound to run into bug fixes or api improvements that will force your hand as well. Not being proactive about keeping your dependencies up to date is like only eating dessert and never eating vegetables.
Being overly proactive is just as bad. The majority of the bugs haven't been in there since the init commit, they were added over time. Often you are just swapping known bugs for unknown bugs (which is arguably worse) particularly when the update contains more than just a tiny bug fix.
Though I concur that your caveats are valid, I do think dependencies should be upgraded to newest-LTS sooner rather than later. Once the dependency falls out of upstream maintenance, it's a time bomb; and we've found (inhouse stuff for public website consumption) that if you don't do the upgrade to newest-LTS sooner, you never quite get around to it until it blows up in your face. And that's not so good.
We keep finding over and over that the wonderfully-engineered project is thrown away after about three years, but that nothing lingers longer than a temporary hack. So we're getting better at futureproofing our temporary hacks ...
Upgrading everything without cause is one of the worst ideas I've heard in a long time. It leads to a lot of unnecessary work that could likely have been avoided otherwise, not to mention breaking the app itself. And if you end up on a version of a dependency that no longer works or is no longer supported in favor of a new package? What then? Rewrite all my code? What incredibly stupid advice to upgrade everything. The rest of the article is not much better either.
If you are using a web framework, its a good idea to keep it up to date, a many security holes get fixed. And in my experience when doing this with Django, its often necessary to upgrade some of the packages so that they work with the newer version of he framework. Keeping everything (reasonably) up to date as you go is the easiest way to avoid "one big upgrade of everything" - which can be a nightmare as you suggest.
The article's focus is on avoid brittle code. By updating your dependencies often, you will shake out accidental coupling or assumptions in your code. It might be expensive in the short time, but it will make future upgrades easier. For example, if you are on library version 2.1 you will have a harder time jumping to version 4.3 to get some headline-making security update. You also avoid time spent debugging bugs that may have been fixed in a newer version of one of your dependencies.
That said, updating every week might be unnecessarily frequent. :)
While the advice seems well-reasoned and complete, I am not sure how the test-before-you-implement philosophy extends to GUI/graphical development.
I work on high-level embedded applications where we design and simulate the application on the development machine before flashing it on the target. The usual test method would be to compare the expected/actual images and check to see if everything matches the requirement.
In this case though, usually it is not possible to test something automatically before implementation since there is nothing to compare the results (images) with.
Can anyone suggest how TDD-like methods work in this case?
The gist is, make sure your UI does nothing other than merely displaying the view model. The business logic can then be tested without having a GUI. Please watch the full video if you are interested in learning more.
There is some good points in there, but there has to be a limit on how many times you can sell unit tests. I prefer advice that goes like: we did it this way and our results were the following, over the you should totally do this generic idea we renamed once again consultant speech.
This is dangerous advice. There's some good advice underneath. Updating often addresses pain points iteratively and is easier than doing major upgrades every year, for example.
But you should not be blindly updating your dependencies every week. It's a distraction and some updates require dev time to fix things (hopefully your integration tests catch some of these).
It's also dangerous if you update all without looking at changelogs. Maybe the new version of one dependency brings in ten new dependencies, or maybe it brings in security vulnerabilities, or maybe it's a completely different codebase or package was removed like what happened with the recent npm fiasco.
Dangerous to do it and dangerous not to do it. The React ecosystem for example is a particularly fast moving fish; I think major releases in backend languages are far easier to be up to date with. I wonder if this applies to infrastructure too; always being on the latest version of Ubuntu for example seems overkill. Having said that, I think I generally agree with the sentiment of this article.
In my experience, rationality is better. Everything we do to our code assets has an associated cost. Therefore, everything we do should have a clear justification -- unless of course you have countless world and time to spend. It would be pointless to upgrade from your old MVC framework if there's no clear and prioritized problem it would solve. Professional developers acquire tools and instincts over time--not blind rules like these--and always use their brain when solving problems.
It's worth noting that this blog post is on the GoCD website; so my expectation is that it's part of a larger theme of making testing upgrades an automated process.
One thing that makes this hard is that there are so many sources of upstream variability these days. On my current project I am up to ten (or 13, depending how you count) sources of upstream changes that I can think of. And this is only a medium-sized project.
My predecessors had 4-5 pairs to work with, so having a chore track humming was a viable tactic, so these could pretty much be handled manually. With 2 pairs and a lot of feature work it's much harder to justify running a chore track every day.
> Everything we do to our code assets has an associated cost.
Indeed. Not upgrading (incrementally in small, manageable chunks) is one of the highest costs to incur. Almost everything in software dev is done is small, iterative chunks for a good reason.
> Not upgrading (incrementally in small, manageable chunks) is one of the highest costs to incur.
Absolutely not true as generalizable statement. Not even close. I've worked on several large software projects, from systems dev and full stack distributed web systems that have ran solidly for years and years on non-latest versions. Our productivity would have severely been hampered by a slavish adherence to a rule like this.
I have often wondered if a total rewrite of an app every few years would take less effort that putting in the work in the beginning to make it upgradeable. This design to make it upgradeable often fails anyway, especially when the team changes.
That's not the first choice you should make. First, a fresh code base is going to have new bugs. Second, you risk the second system syndrome (https://en.wikipedia.org/wiki/The_Mythical_Man-Month) where you over engineer it, make a bunch of new decisions, and may never ship.
Not to say that it isn't sometimes time to throw the old infrastructure away, but it's not a decision you take lightly.
Similarly Working Effective with Legacy Code by Michael Feathers (http://amzn.to/1UxwVdL) is a great programming book. I appreciate it because It's really nothing but patterns for dealing with bad code (mostly Java, but most of it translates to other languages). Very little why (which I already know), lots of "how to fix X", aka, great signal to noise ratio.
Software development is not a ponzi scheme. Employees that sabotage new employees suck.
Making yourself 10x more productive than the average by making everyone else 0.1x as productive as you is not in the best interest of the company.
Feeding a self-perpetuating circle of unethical sabotaging losers is not in the best interest of the company. It's cancer that needs to be extirpated as soon as possible and made an example of by firing them to the sound of a trumpet.
Hire people that see their job as something more than a paycheck. Hire people that care.
Hint: people that talk about their weekend, past-times, vacations, cars, etc., and change topics or moods when engaged with work related challenges, are people that are most likely checked out. Those are the people that won't care and will drown your company in their ego.
What you described at the end is a perfect middle manager Machiavelli, who "works" by sitting at their desk all day to make others look bad.
Either that or they are a soulless drone who hates all people around them and because of their excessive (uncreative and diminishing returns be damned) obsession demands untoward shares of the "rewards" and is surrounded by "morons". These people also worship Ayn Rand.
94 comments
[ 1.9 ms ] story [ 167 ms ] thread> And—yes—you can read this as my being unlikely to use Rails the next time I'm building a traditional multi-page application.
Really? The way I do it is I write a client library for the API and run the tests in Rspec on top of the client library. Sure I'm going through the whole stack, but the thing operates as fast as you can serve normal users! Surely your large project can handle 5000 concurrent requests? While that might seem like a lot of buildout (scalable testing servers), I'd argue the opposite is a lot of buildout: writing an application without Rails.
Granted my approach might be most feasible for Rails API + Ember, but I really don't see the speed of my tests passing as what's holding me back.
The other thing that is nice is that once you start needing to speed up endpoints by pulling them into a compiled language, you can keep your tests! Everything is going through the API anyway! So feel free to Nim or Rust the CPU heavy endpoint or Redis-cache the write-once-in-a-while-and-keep endpoint.
Some companies invest in huge infrastructure to speed things up (paralleling until you can't...) but in that situation, suddenly Devs can no longer build locally without biting their tongue.
Setting up end-to-end test "per test-case" is also tedious (should user be logged-in in order for this test-case to run? what should be the state before the tests are run and what should be the state after the tests are run? what if the state of the system have the expected _and_ more e.g.: side-effect?)
Why not? This is a simple helper function or easily parallelizable block. Same with creating a sub-resource before modifying it. People act like this is the end of the world, but it isn't. And if you like, you can still do your (in my controversial opinion) silly unit tests and you can still parallelize those too!
Whether a dev can build a project locally or not does not relate to whether or not the tests can be run in parallel.
1. Which user? which role should the test written as?
2. Suddenly bug showed up because expectation between roles differ (usually the bug is in the test framework)
No, it's not the end of the world, just programmer's productivity go down...people can still work for the project just isn't fast as it used to be anymore...
As for programmer productivity, proper parallelization of these tests results in sub second test suite runtime. Maybe more if there is a new dependency that the testing servers need to install, but it really isn't that bad.
That's why you pick a few smoke tests to run before checking the code in and let the full regression test suite run while you're doing other stuff.
>Setting up end-to-end test "per test-case" is also tedious (should user be logged-in in order for this test-case to run? what should be the state before the tests are run and what should be the state after the tests are run? what if the state of the system have the expected _and_ more e.g.: side-effect?)
You need a decent integration testing framework to help here.
Ideally it should be refactored right away as you say, but often it takes forgetting the code, then rereading it to see how unintuitive it was the first time.
In a similar vein, I like to take a break if possible, before a larger commit, then come back to it and see if I should refactor first.
Two benefits of this are (1) a fresh mind to see what you might have missed especially since when I'm done with a piece of functionality I often just want to commit it and move (2) it gets me back in the flow of development quickly.
I'm not saying that unit tests aren't useful. They definitely help for example if you're using a dynamic programming language or if you're practicing TDD. I would just emphasize a higher priority for integration and acceptance tests.
It can feel a little dangerous, but as long as the integration and e2e tests are kept up and run on build, i think I'm covered. I've definitely been bitten before, but I've been bitten more by over complicated and long running tests.
I find both Integration and Acceptance Tests could potentially hide many bad internal designs (tight coupling, etc). I also find that not many places can speed up their full integration/acceptance tests suites; suddenly build becomes way too painful, be it local dev build or CI-triggered-through-commit builds.
Note: My experience with super-slow integration tests choking build is with a large JavaScript project with 100 developers (at least).
I implemented integration-tests for a Java-heavy based project before but limited only to test the database component of the project (e.g.: Repository, JPQL, JPA), the ability to test business logics that are tightly coupled with ORM quickly in isolated manner is a blessing.
I guess YMMV.
They must be testing "behavior". If you are refactoring to clean up code, you'll want to verify the behavior and unit tests gives you that quick feedback.
An analogy is that unit tests are like a brick layer's spirit-level and plumb-line. As you build the wall, you ensure the pieces are put together correctly and head off mistakes early.
The "only test at a high-level" is like saying don't confirm the bricks you are laying are correct, instead bring in an inspector to check the whole building is up to code once we have finished.
If we do bring in the inspector she takes one look and says the wall have to come down because of faults right at the bottom or because we forgot the damp course or we never get that far because we couldn't fit the roof because all the walls bowed out.
I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases and that they are not in a position to be advising others.
And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time.
Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with.
Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?
I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite.
On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense).
Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help.
However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...)
(I know this is a very short answer but I'm typing on a phone)
What? On the codebase I work on, integration tests run every single commit, just like unit tests. They would both fail at the same time. Except the integration tests are much less brittle.
The faster your test suite, the more often you can run it, and the faster you get feedback on your code.
You can combine this with a healthy integration test suite, and only run the unit tests.
Validation rules is a good example, if I want to test that my user input validation rules are correct, I don't need to do this through the entire application frontend - I can just check those rules in isolation.
In all likelihood, the faster it is, the more tightly coupled it will be. You're simply exchanging computing power for brain power (refactoring tests when you refactor code), which is nearly always a bad trade off.
In most cases I want to preserve existing behaviour, so I don't want any existing tests to fail.
I'll add tests for the new behaviour in the relevant modules, add code to make them pass, and then if necessary update the glue code / integration tests which cross the boundary.
Pretty much by definition, a refactor of code should not change its behaviour. My tests usually assert on behaviour observed at the boundaries of modules, so they should still pass during refactorings.
Quite precisely by definition, perhaps for other reasons as well :-P
> My tests usually assert on behaviour observed at the boundaries of modules, so they should still pass during refactorings.
I find the kind of refactoring that most desperately needs the support of tests is changing the boundaries of the individual units. Unit tests can still arguably be useful there in making me clarify the changes I'm making to my interfaces... but they do need changing, and I can't be quite as confident that nothing is breaking if I'm changing both code and tests in tandem.
I'm with GP about integration tests.
Sometimes I'll wait until I've finished changing all those function signatures until I hit save.
That's funny. I suspect that people who advance the pro-unit test view do so haven't had sufficient experience with large complex code bases.
Experience with large complex code bases with a spaghetti problem is the exact thing that put me off unit tests.
Unit tests are by their nature very tightly coupled to the underlying implementation, which is why they cause so many problems.
It's kind of like pouring concrete over a bunch of code bricks: if those bricks are in the wrong place and need refactoring, by pouring unit tests over them means you're solidifying the broken architecture and making it even harder to fix.
IME there basically don't exist large projects without technical debt problems either, so arguing that unit tests work well when the architecture is good is kind of a moot point.
> Unit tests are by their nature very tightly coupled to the underlying implementation
No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. It shouldn't care about the internal behavior of the "unit". That's the theory. They're only supposed to be coupled to the interface, not to the implementation.
I've found this to be one of the biggest myths of unit test driven development. Developers who have a tendency to write spaghetti do so anyhow. When it becomes too hard to write tests they simply don't write tests or they write 'pretend' tests.
It's perfectly possible to write spaghetti code and write a badly written test that notionally covers it.
Good coders code well. Bad coders don't. TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation.
>That doesn't help you if you are adding tests to an existing project that's a mess, though.
Harms you deeply, in fact.
>No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior.
i.e. it couples itself to a unit, which is an underlying implementation detail.
If it's coupling itself to an externally facing interface it's an integration test or an end to end test.
>It shouldn't care about the internal behavior of the "unit".
No, but then again, since "units" are supposed to be the smallest blocks of code that comprise your code base, it's caring very much about how all of the blocks talk to one another, which is an implementation detail (for your project). Perhaps the most crucial implementation detail, in fact.
How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt.
Pouring test cement in those cracks means that any technical debt accrued in those API contracts becomes impossible to refactor without breaking the tests.
In practice this makes that refactoring more difficult than if it had no tests at all due to the massive psychological inhibition against deleting tests or turning the build red and the added problem that real failures look the same as changed API contracts.
Absolutely.
> Good coders code well. Bad coders don't.
Also true.
> TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation.
I've used TDD and not used it. And when programming using TDD, my code was less "spaghetti" - more modular, better-designed interfaces. There definitely was causation, in my case - it made me write my non-test code differently. So that's anecdote and not data, but holding the programmer constant and changing the methodology, the style and quality of the code changed.
> >No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior.
> i.e. it couples itself to a unit, which is an underlying implementation detail.
> If it's coupling itself to an externally facing interface it's an integration test or an end to end test.
Let's get specific, so that our terms can be better defined, and we can make sure that we're not just arguing about definitions of terms. Instead of "unit", let's say "C++ class". Now: By my definitions, my unit test should test the behavior of the public API of that class, not the private implementation. So the API is an implementation detail of the project, but not of the class. I will agree that the unit test definitely couples to the interface of the class. Done right, though, it doesn't couple to the implementation of the class.
> How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt.
A bigger source of technical debt than the bugs you can't find that are buried in the implementation of some class? I doubt it. And good tests mean that those bugs don't make it very far before getting exposed.
A bigger source of technical debt than that API that you don't know how to use, and so you call it wrong? Tests can actually serve as documentation for how to use an API. You have non-test documentation? You're lucky to have that, but the problem is, when the code implementing the class changes, the documentation might well not. If that happens with the tests, you find out the first time you run them.
In almost all cases (the exception being if the class is the core of a library's externally facing API), the class itself is an implementation detail.
>A bigger source of technical debt than the bugs you can't find that are buried in the implementation of some class?
Yes, obviously. Bugs are not technical debt.
>And good tests mean that those bugs don't make it very far before getting exposed.
And integration tests are generally better at finding said bugs than unit tests since they test more realistically.
>A bigger source of technical debt than that API that you don't know how to use, and so you call it wrong?
Is again, a bug - not technical debt.
Unit tests are horrible for this since they require you to mock APIs that you're calling. The test which mocks the API wrongly won't catch that bug but the integration test that uses the real API will.
>Tests can actually serve as documentation for how to use an API.
Yes they can. Integration tests can double as a specification for an entire application which is even better than a documentation for a (potentially) badly architected API.
Both, unit and integration tests, are imperative. Integration tests point in the general direction of where an issue might be but unit tests are more precise.
1. As your integration tests will both more numerous and complex to cover corner case scenarios, the time it takes to execute the suite will grow too. And since the integration test is more opaque too, this price has to be paid by both you and all the developers that touch something that might have used your module.
2. The time at which you receive the feedback of the test is already too late. You are no longer doing experimental changes, you are trying to push this unit of work into the code base and be done with it. Either you will be tempted to gloss over any divergence you find (quality price) or you will have to execute some relevant subset of your integration testsuite during your coding/OODA loop (execution speed price).
Choosing unit tests because of speed is nearly always a premature optimization.
IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode.
Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).
Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play.
A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.
However, a lot (perhaps most) code is more about integrating disparate components (browsers, lots of different kinds of external APIs, databases, times, distributed task processors, etc.) rather than lots of complex logic and for that kind of code, unit tests are substantially less effective and more horrible to use whereas integration tests really shine.
I think ideally, you should keep the two kinds of code strongly segregated, with clean APIs between them, so you can easily use the appropriate kind of testing for either one.
I don't agree that a good code base necessarily has both types. A web app with relatively shallow logic and lots of integrations only really benefits from having integration tests. A library to convert human readable strings into dates likewise gains nothing from having integration tests.
Similarly, a messy spaghetti codebase with a disgusting mix of both types of code can only really benefit from integration testing until it's substantially refactored and the two types of code are segregated.
Why _strongly_ segregated?
Nothing stops you to write unit tests for the methods that should be covered by unit tests and integration tests for the functionality that needs to be covered by integration tests.
How would code segregation help?
That test will also often break as you refactor the various connecting APIs it surrounds.
Tests that break in the presence of changed API contracts (as opposed to bugs) are vastly less valuable than tests which only go red in the presence of a bug.
I normally structure two type of integrations test: 5-10 minutes tests that run before every checking.
With those test coverage in place. The worst time I receive the feedback from the test failures are one day or one weekend. All the changes are all within my mind that might cost the failure.BTW, ALL checkin code should be passing ALL integration tests. If they don't, the highest priority is fixed the test failures and not develop any new feature.
No it's not. Not at all. The whole point is you can change the guts of a unit around and know you haven't broken things cause the unit tests still pass. Because the unit test is testing the "contract" (explicit or implied) the unit has made with other units about it's inputs and outputs.
Unit test should fail (and require updating) only if the unit's INTERFACE changes.
But, if your unit tests are failing everytime you touch your code then 1) your code is not cleanly isolated and independent and/or 2) you have poor unittests.
You only need that sort of test in a programming language or public library, because you're publishing the behavior to unknown numbers of users, and it has to work, as documented. If they find too many things that don't work according to the documentation, they will correctly suspect that you aren't testing, and lose confidence.
But if some API is internal to an application, and no code depends on some aspect of its interface contract, then that aspect effectively doesn't exist.
It may happen that someone discovers this aspect in the documentation and tries using it. At that time it is either found broken, or found working. If it is found broken, it can be fixed. Or else it can be striken from the documentation (and another way can be found in the application code to meet its requirements).
If the aspect is found correct, or fixed to reflect the documentation, and is then used in the application, then that use constitutes a test. The feature is tested indirectly, through some regression test case which targets something in the application, which depends on that aspect of the API contract lower down.
If you design units with all kinds of detailed aspects in their interface contracts which end up not being used at all, and test all of them, it can be argued that you're wasting time: you're basically working on your library design hobby, at the expense of the project, which just serves as a vehicle for it.
"Can't tell unless" means exactly the same thing as "can tell". You have a way to get to that corner case through the public functionality.
If there really is no way to tell through the current public API, there the bug doesn't exist.
Are you arguing that all unit tests should be black-box?
Refactoring is defined as the process by which you restructure code without changing its external behaviour. So in other words, refactoring should never break tests because the external features of the code do not change.
Now - redesign or rewriting code is a very different thing. Refactoring can lay the groundwork for redesign or component level rewrites. But the process of redesign and rewrites could result in breakages unless the right level of isolating layers are put in place. This usually involves lots of work to keep tests working.
Personally I have found that unit tests don't result in less work. They are a LOT of work and will continue to be a lot of work if they are to generate value. What they offer though is the ability to know when things have gone wrong and the confidence to release and deploy. I realize that you don't get increased confidence in your release process by doing less work. Software it appears, like almost everything else, obeys the basic laws of thermodynamics. :)
And what are those "things" that haven't broken? Why, unit tests.
As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported.
The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well.
To the weaknesses of the article with regard to its critique of end-to-end tests:
> Let's assume the team already has some fantastic test infrastructure in place. Every night:
Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests.
> To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete.
It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to:
> A failing test does not directly benefit the user.
A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied!
> End-to-end tests were flaky at times.
Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests.
Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests.
It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.
Flaky tests are indicative of bugs in your tests.
Google is literally advocating treating your bugs as a fact of life, not fixing them, and avoiding harder engineering challenges because they are harder.
This is the worst possible advice.
Integration and functional tests, on the other hand, are long-lived tests intended to ensure that all the pieces stay glued together. They allow the middle parts of code to squiggle quite a bit, as long as they get to the same place at the end of the line. They allow for safe refactoring of modular units whose entire scope lies beneath the boundary of the test. And when the functionality of the application or library changes, they should change as well, but this should be an order or two of magnitude less frequent than the unit test level.
As a rule of thumb, I put integration tests around any piece of functionality that "makes me money"—less literally, around any bit that, if it were broken, would prevent things like user signups, users paying the correct amount, or critical functionality that is indispensible to a user of the software. Likewise, I usually write unit tests when i'm writing new modules so that I don't have to keep reloading a browser or setting up complex data schemes while they're in development. Sometimes I remove those, especially for code that is low-churn (which is a totally under-appreciated metric to look at when you're figuring out where you should focus coverage, IMHO.)
Both types are important necessary. And not all of them are destined to live forever.
Here's the thing about "agile" and Unit Tests. The current generation of agile was re-discovered in Smalltalk, after the Smalltalk Refactoring Browser had arrived on the scene. Not only could you refactor at warp speed and high confidence (1) and with unlimited undo, since Smalltalk SUnit test were just ordinary Smalltalk code, Unit tests often got refactored right along with the code.
If you're in an environment where you have to painstakingly refactor, then also separately update your Unit Tests, then it's going to feel like you're doing a whole lot more work -- because you are!
Given that very few environments have the requisite properties, TDD is going to feel like a slog on most stacks, and people will naturally gravitate to something lighter, like integration/acceptance/smoke tests.
(1) - Yes, the prepackaged Refactorings in the original Smalltalk RB were guaranteed to be correct. There were sophisticated runtime checks, and they'd refuse to fire off if correctness wasn't completely certain. You could fool them -- because dynamic language -- but you had to try pretty hard to mess up like that. Then the rest of the shop got really annoyed at you!
I'm thinking about third party dependencies here (eg via package managers). In my opinion it's best practice to stick with one certain stable version and then stop updating it - to avoid wasting time on possible needed refactoring due to changed dependency behaviour. Monitoring the change log of the dependencies to see if security issues are fixed and reacting on that is of course still reasonable. Yet upgrading just because " upgrade as often as possible" is not.
Maybe it's different for projects lasting 10years or more, maybe there are other rules to make. My experience is with 2.5yr-projects in average.
Maybe I'm working with dependencies which get updated every week or so, maybe the author's dependencies are updated every half-a-year.
Moving an expense from today to the future is debt.
Debt (technical or otherwise) in itself is not a bad thing if used sensibly for some gain. Allocating resources to maintain dependencies is not necessarily the best short or long term investment.
These are all heuristics.
If you're building a traditional piece of software (no "evergreen" continual updates, maybe even with no upgrade process at all eg. device firmware) then just using a library internally, and your requirements aren't changing, then continually updating it is busywork at best, and a risk to the reliability and stability of your project at worst.
It's not "technical debt" to skip point releases that don't add anything to your project, since you avoid potentially re-integrating changes to the same features if they change multiple times, and there's a good chance you'll never need to update at all.
In addition I've found the stability of the system as a whole is better when you swallow the pie in small chunks and when your project depends on multiple dependencies upgrading them gradually piece at a time makes bisections much simpler than "stop the world and update".
It really does depend (as most things do) on the particular domain/problem/solution though, if I was working on something that ran for years and couldn't be touched in production I'd probably go the other way.
We keep finding over and over that the wonderfully-engineered project is thrown away after about three years, but that nothing lingers longer than a temporary hack. So we're getting better at futureproofing our temporary hacks ...
That said, updating every week might be unnecessarily frequent. :)
I work on high-level embedded applications where we design and simulate the application on the development machine before flashing it on the target. The usual test method would be to compare the expected/actual images and check to see if everything matches the requirement.
In this case though, usually it is not possible to test something automatically before implementation since there is nothing to compare the results (images) with.
Can anyone suggest how TDD-like methods work in this case?
The gist is, make sure your UI does nothing other than merely displaying the view model. The business logic can then be tested without having a GUI. Please watch the full video if you are interested in learning more.
This is dangerous advice. There's some good advice underneath. Updating often addresses pain points iteratively and is easier than doing major upgrades every year, for example.
But you should not be blindly updating your dependencies every week. It's a distraction and some updates require dev time to fix things (hopefully your integration tests catch some of these).
It's also dangerous if you update all without looking at changelogs. Maybe the new version of one dependency brings in ten new dependencies, or maybe it brings in security vulnerabilities, or maybe it's a completely different codebase or package was removed like what happened with the recent npm fiasco.
In my experience, rationality is better. Everything we do to our code assets has an associated cost. Therefore, everything we do should have a clear justification -- unless of course you have countless world and time to spend. It would be pointless to upgrade from your old MVC framework if there's no clear and prioritized problem it would solve. Professional developers acquire tools and instincts over time--not blind rules like these--and always use their brain when solving problems.
One thing that makes this hard is that there are so many sources of upstream variability these days. On my current project I am up to ten (or 13, depending how you count) sources of upstream changes that I can think of. And this is only a medium-sized project.
My predecessors had 4-5 pairs to work with, so having a chore track humming was a viable tactic, so these could pretty much be handled manually. With 2 pairs and a lot of feature work it's much harder to justify running a chore track every day.
Indeed. Not upgrading (incrementally in small, manageable chunks) is one of the highest costs to incur. Almost everything in software dev is done is small, iterative chunks for a good reason.
Absolutely not true as generalizable statement. Not even close. I've worked on several large software projects, from systems dev and full stack distributed web systems that have ran solidly for years and years on non-latest versions. Our productivity would have severely been hampered by a slavish adherence to a rule like this.
Not to say that it isn't sometimes time to throw the old infrastructure away, but it's not a decision you take lightly.
But a major cause of that syndrome is adding new features to legacy code. If the code was new that wouldn't be a factor.
Similarly Working Effective with Legacy Code by Michael Feathers (http://amzn.to/1UxwVdL) is a great programming book. I appreciate it because It's really nothing but patterns for dealing with bad code (mostly Java, but most of it translates to other languages). Very little why (which I already know), lots of "how to fix X", aka, great signal to noise ratio.
Feeding a self-perpetuating circle of unethical sabotaging losers is not in the best interest of the company. It's cancer that needs to be extirpated as soon as possible and made an example of by firing them to the sound of a trumpet.
Hire people that see their job as something more than a paycheck. Hire people that care.
Hint: people that talk about their weekend, past-times, vacations, cars, etc., and change topics or moods when engaged with work related challenges, are people that are most likely checked out. Those are the people that won't care and will drown your company in their ego.
Either that or they are a soulless drone who hates all people around them and because of their excessive (uncreative and diminishing returns be damned) obsession demands untoward shares of the "rewards" and is surrounded by "morons". These people also worship Ayn Rand.