I quite like the distinction this article draws, and believe it’s important, but I have had only limited success in trying to get coworkers to go along with it. Everyone seems to use mock, stub, and fake interchangeably, even if they’re using a specific mock library.
I’ve read this article twice now and it feels like confused word soup with no clear statements of the intended differences, specifically none of the statements about the order of state vs behavior validation are intelligible to me. I have read them multiple times and just feel (a) frustrated and (b) a mock and a stub are really the same idea with nothing more than occasional conventional differences that amount to semantic hair-splitting with no material consequence or need for articles to be written about the platonic difference.
What do you mean by “order” in the sentence “... the order of state vs. behavior validation ...”?
State validation means checking a stub dependency of the SUT to make sure it’s in the right state. Behavior validation means you check a mock dependency to make sure the right functions were called the right number of times, and in the right order.
A stub is just a thing you can substitute for the dependency that knows how to tell the test it’s state, and a mock is a substitute for the dependency that knows how to tell it’s behavior, like which functions are called and how many times.
In terms of them being the same thing, they are both types of Test Doubles, so they both fall into a higher level category of “a substitute for a dependency of the system under test (SUT)”. The difference is how they’re used.
I guess in the example given you could also validate behaviour by using a Spy Warehouse (a capturing stub).
The theoretical distinction with Mocks is that they contain their own assertions. The practical distinction (when using mocking frameworks) is often that they allow testing a SUT that depends on concrete classes, without subclassing them.
The same "final state" of a fake can potentially be achieved by many different patterns of interaction between the system under test and the fake.
We don't really care about those patterns of interaction as long as they end up doing the right thing: allowing the SUT to affect his "outside world" as intended.
Making assertions about the exact pattern of interaction (that is: using mocks) is too high granularity, and will make the tests break on valid refactorings. Because we don't care about the pattern itself—it's just an implementation detail—but about what it achieves.
Worth noting is that this article is almost 15 years old so the "need" for this article probably was a bit different back then.
That said, much of this still is valuable to think about and this article, to me, is not just word soup but I'll give you that it's not very focused. And I get confused since "mockist tdd" isn't really talked about explicility anymore. But I would argue the concept is very much alive as a natural part of behavior-driven development which encourages outside-in development with a focus on writing acceptance test first. So my question is if Fowlers worry from the past, that using mocks and asserting behaviour is focusing too much on the implementation. But like Fowler I haven't done BDD "for real" so I haven't seen how it plays out in the long term.
I found quite useful an example I read in the book "The Art of Unit Testing" by Roy Osherove, it was something like this (I don't have it at hand, I repeat it from memory):
Suppose you have to test the irrigation system of a tree in your garden, you want to test how many times and how much water is used, you have two ways to do it:
1) State-based: run the system for a few hours, during which it should water the tree several times. At the end of that time, you check the state of the watered tree: Is the soil moist enough, is the tree well, are the leaves green, etc. (Obviously this is just an example as this test would be complex in real life, but it is to get the idea across).
2) Interaction: We place a flow meter, a device that registers how much water is flowing through the device and at what times. At the end of the day, check that the device has been called the correct number of times, with the correct amount of water each time, and we don't worry about checking the status of the tree.
>I’ve read this article twice now and it feels like confused word soup
Yeah, that'll happen with a lot of the testing "literature" from the 2005ish to 2015ish era.
It was a popular topic at the time, and one I was searching for answers on too. But I was working at a company that was ahead of the curve on testing and I got so frustrated at publicly available resources on testing. During that time and still to this day, when I read about testing online I kinda feel like I'm drunk. I have an idea of what's going on, things sorta make sense but not really. I'm only understanding bits and pieces. There are unnecessary and over-complicated details on some things, suspiciously light on meaty topics.
There was so much drivel written about testing back then that I still get angry remembering that people would say with a straight face "you don't need documentation, tests are your documentation."
I more understand the Martin Fowler article quite well, but only after struggling with it and revisiting it (sporadically) over several years, and I really think this by Mr, Khorikov gives a much clearer, more straightforward overview.
I disagree. I've never been in a conversation where using any of these terms led to a misunderstanding. Everyone understands that mocks, stubs, fakes are programmer-written stand-ins for other objects.
In the vast majority of the cases no one cares whether it returns a preset value or only counts the number of hits or whatever else it might do. And if that level of implementation detail becomes important, then it's described that way, not by the use of Fowler's terminology.
The shame of it is that the title of this essay makes it appear like it's simply an introduction to a lexicon, whereas the word definitions are actually a small part of the essay and there are more important topics presented (and presented fairly well).
>Everyone understands that mocks, stubs, fakes are programmer-written stand-ins for other objects.
Sure, but that's only the starting point here.
>The shame of it is that the title of this essay makes it appear like it's simply an introduction to a lexicon, whereas the word definitions are actually a small part of the essay and there are more important topics presented (and presented fairly well).
Yeah, but if you can't use the words to talk about these topics, it's going to be a lot harder to do so without getting verbose.
Fakes not Mocks- I hopped on this train and am super happy.
I start out with a small & simple fake for something as I'm working on the layer above it, then keep flushing it out as I move on to higher and higher layers.
In the end I have a suite of completely customizable fakes for all of my external dependencies that I can tweak to my every need.
I have tried using fakes over mocks, but it becomes cumbersome if it is a large-ish code base with handful of developers contributing to it. Keeping expectations on what a particular Fake does gets contentious. Mocks however usually allow a developer to encapsulate entire mock behavior within the boundaries of a unit test.
I am curious to know if you ran into similar problem, if you did how you managed to overcome it?
This seems like an odd assertion to make, since fakes and mocks have largely non-overlapping purposes. Fakes provide some simulacrum of real behavior, while mocks just verify that appropriate messages were sent to the thing they're standing in for.
I do see fakes as having overlapping purpose with stubs, though. But, like the title of the article says, mocks aren't stubs.
I find the distinction between “fake”, “stub” and “mock” very useful when making decisions about my own code, but I’ve basically given up on using it to communicate because I never find myself dealing with anyone who cares about the different meanings. Most people I know use “mock” to refer to any kind of test double.
Don’t give up :) ‘mock’ has indeed become an umbrella term for all test doubles, but the other terms still have specific technical meanings, so it’s still worth using them correctly.
If you’re working with technical people who don’t care about technical details then that’s probably not a good sign!
So, first off, a disclaimer: I'm a partisan of classical TDD.
With that out of the way, I would recommend using a test double in exactly one kind of situation: When it's genuinely awkward to use the real thing. Concrete examples include talking to a third-party API, testing for error conditions that are difficult or impossible to reliably organically produce, and when the real dependency hasn't actually been implemented yet.
As for the different kinds of test doubles, Fowler has a concise overview at https://martinfowler.com/bliki/TestDouble.html that is just about the clearest explanation I've seen of the different kinds and what they're used for.
> With that out of the way, I would recommend using a test double in exactly one kind of situation: When it's genuinely awkward to use the real thing.
THANK YOU.
I'm actually pretty staunchly anti-mocking (and anything similar). I think too many developers start thinking of testing as the goal rather than as the means. What this means to me is I don't care if it's technically an integration test vs a unit test, I care that it tests _REALITY_.
There are times when the isolation given by mocking can be useful, but I submit it's useful far far less often than many claim. And your examples are a good use as well.
But imo trying to mock everything is mostly mental masturbation only with long-term maintenance costs.
Classic article, and important distinction. I’ve found that those who aren’t familiar with the distinction tend to be less aware of what and how they’re testing. Lack of awareness or understanding tends to lead to thing like excessive mocking.
Poorly named libs (e.g. mockito) just exacerbate this problem.
Hmm, I'll bite.
What exactly in your opinion makes the difference here with awareness or lack thereof (of the distinction)?
The way I see it mocks are the all encompassing tool here. With mocks you can both stub behaviors and, if you want, verify behaviors.
And in reality every mock needs to act as a stub, e.g. a mocked method returning a preset value.
So, it seems natural that "mocking" as more encompassing concept is what is used in library names. If people end up using those to mostly create stubs instead of mocks so what?
My experience so far has been that people naturally flock to a solution for testing that is flexible and lets them quickly cover the boundaries of the code they are testing. If people want to check if their code calls a method x times but are not interested in what that method does for the test, they use mocks. If they do they use "the real thing" or a fake on which they can validate state later. And if they just need a method to return something and don't care about whether it's called or how often they use a stub.
I feel that often when you stub you can also go ahead and add code to verify some behaviors (mock) to get some extra coverage. I don't know I just don't think it's bad to mix and match if it gets you to more things tested.
My experience with junior (and sometimes not junior) developers has been that the lack of understanding leads to confusion about what they're trying to test. I think the ambiguity leads to a tendency toward mocks, which leads to tests that are focused on collaboration and implementation rather than output. Sometimes that makes perfect sense, but I've also seen where it encouraged more tests that are too implementation reliant.
Nothing wrong with mocks of course, sometimes they are absolutely required. I just favor isolation using stubs, and using mocks sparingly and only when absolutely needed.
> ...in reality every mock needs to act as a stub
I disagree. In fact a collaborator method call that doesn't return a value is a great candidate for mocking. Conversely, a collaborator that does return a value can usually be stubbed.
> ...when you stub you can also go ahead and add code to verify some behaviors (mock) to get some extra coverage
Entirely dependent on context of course, but to generalize, this feels risky. In many cases adding extra coverage via mocks might be making tests more brittle, for little to no value gain. Further if every method you mock is returning a preset value, there's no reason to verify expectations on it, because it's a stub that will allow verifying expectations on the output of the object/function under test.
As far as library naming, it's just a pet peeve stemming from "mock" somehow becoming synonymous with "test double". If people use a "mocking" library for stubs... well... that's good! :)
In my experience, I have never seen people accidentally (or ignorantly) wire a stub to a stub. When you do this it is really obvious, because there are only trivial, or maybe no, assertions possible.
I have often - like on a team of 5-10 at least 2x per year - seen people wire a mock to a mock or a fake to a mock, and either not realize it (because it's under six layers of DI) or just they're not paying attention but someone said they "have to write tests" and trying to get it over with. And this is generally a lot less obvious, because the test suite is still full of assertions, sometimes even totally reasonable-looking ones. (Such tests are even sometimes "real tests" - but that the mock is doing what's expected, not of the desired SUT.)
There's also the general argument from the principle of least power. The less code in your doubles, the more certain you are the doubles are doing what you expect, and the more you're verifying what really matters in your code. Like how pure functions are preferable to impure functions when possible, having comprehensive tests without concerns about "side effects" also means your code has fewer relevant side effects.
I use a mocking library mainly to produce stubs. I found verifying mocks too brittle in many cases. Generally I want to test outcomes (state), not behavior. I had many changes breaking tests although functionality was fine.
The example outlined in the article isn't exactly a unit test: The change in state of one object changes the state of another object. If these are coupled that tightly I'd rather try to mock (stub) the dependencies of both objects and let them interact (test both at the same time), then verifying their final state.
There is what I consider to be an important caveat near the end:
In essence classic xunit tests are not just unit tests, but also mini-integration tests. As a result many people like the fact that client tests may catch errors that the main tests for an object may have missed, particularly probing areas where classes interact. Mockist tests lose that quality. In addition you also run the risk that expectations on mockist tests can be incorrect, resulting in unit tests that run green but mask inherent errors.
29 comments
[ 2.5 ms ] story [ 78.1 ms ] threadStubs just return preprogrammed data.
Mocks do the above, but can provide their own assertions about call order or number of times called.
State validation means checking a stub dependency of the SUT to make sure it’s in the right state. Behavior validation means you check a mock dependency to make sure the right functions were called the right number of times, and in the right order.
A stub is just a thing you can substitute for the dependency that knows how to tell the test it’s state, and a mock is a substitute for the dependency that knows how to tell it’s behavior, like which functions are called and how many times.
In terms of them being the same thing, they are both types of Test Doubles, so they both fall into a higher level category of “a substitute for a dependency of the system under test (SUT)”. The difference is how they’re used.
The theoretical distinction with Mocks is that they contain their own assertions. The practical distinction (when using mocking frameworks) is often that they allow testing a SUT that depends on concrete classes, without subclassing them.
We don't really care about those patterns of interaction as long as they end up doing the right thing: allowing the SUT to affect his "outside world" as intended.
Making assertions about the exact pattern of interaction (that is: using mocks) is too high granularity, and will make the tests break on valid refactorings. Because we don't care about the pattern itself—it's just an implementation detail—but about what it achieves.
That said, much of this still is valuable to think about and this article, to me, is not just word soup but I'll give you that it's not very focused. And I get confused since "mockist tdd" isn't really talked about explicility anymore. But I would argue the concept is very much alive as a natural part of behavior-driven development which encourages outside-in development with a focus on writing acceptance test first. So my question is if Fowlers worry from the past, that using mocks and asserting behaviour is focusing too much on the implementation. But like Fowler I haven't done BDD "for real" so I haven't seen how it plays out in the long term.
Suppose you have to test the irrigation system of a tree in your garden, you want to test how many times and how much water is used, you have two ways to do it:
1) State-based: run the system for a few hours, during which it should water the tree several times. At the end of that time, you check the state of the watered tree: Is the soil moist enough, is the tree well, are the leaves green, etc. (Obviously this is just an example as this test would be complex in real life, but it is to get the idea across).
2) Interaction: We place a flow meter, a device that registers how much water is flowing through the device and at what times. At the end of the day, check that the device has been called the correct number of times, with the correct amount of water each time, and we don't worry about checking the status of the tree.
Yeah, that'll happen with a lot of the testing "literature" from the 2005ish to 2015ish era.
It was a popular topic at the time, and one I was searching for answers on too. But I was working at a company that was ahead of the curve on testing and I got so frustrated at publicly available resources on testing. During that time and still to this day, when I read about testing online I kinda feel like I'm drunk. I have an idea of what's going on, things sorta make sense but not really. I'm only understanding bits and pieces. There are unnecessary and over-complicated details on some things, suspiciously light on meaty topics.
There was so much drivel written about testing back then that I still get angry remembering that people would say with a straight face "you don't need documentation, tests are your documentation."
I more understand the Martin Fowler article quite well, but only after struggling with it and revisiting it (sporadically) over several years, and I really think this by Mr, Khorikov gives a much clearer, more straightforward overview.
In the vast majority of the cases no one cares whether it returns a preset value or only counts the number of hits or whatever else it might do. And if that level of implementation detail becomes important, then it's described that way, not by the use of Fowler's terminology.
The shame of it is that the title of this essay makes it appear like it's simply an introduction to a lexicon, whereas the word definitions are actually a small part of the essay and there are more important topics presented (and presented fairly well).
Sure, but that's only the starting point here.
>The shame of it is that the title of this essay makes it appear like it's simply an introduction to a lexicon, whereas the word definitions are actually a small part of the essay and there are more important topics presented (and presented fairly well).
Yeah, but if you can't use the words to talk about these topics, it's going to be a lot harder to do so without getting verbose.
I start out with a small & simple fake for something as I'm working on the layer above it, then keep flushing it out as I move on to higher and higher layers.
In the end I have a suite of completely customizable fakes for all of my external dependencies that I can tweak to my every need.
I am curious to know if you ran into similar problem, if you did how you managed to overcome it?
I do see fakes as having overlapping purpose with stubs, though. But, like the title of the article says, mocks aren't stubs.
If you’re working with technical people who don’t care about technical details then that’s probably not a good sign!
When would you recommend this practice?
And which of the three would you recommend for what?
With that out of the way, I would recommend using a test double in exactly one kind of situation: When it's genuinely awkward to use the real thing. Concrete examples include talking to a third-party API, testing for error conditions that are difficult or impossible to reliably organically produce, and when the real dependency hasn't actually been implemented yet.
As for the different kinds of test doubles, Fowler has a concise overview at https://martinfowler.com/bliki/TestDouble.html that is just about the clearest explanation I've seen of the different kinds and what they're used for.
THANK YOU.
I'm actually pretty staunchly anti-mocking (and anything similar). I think too many developers start thinking of testing as the goal rather than as the means. What this means to me is I don't care if it's technically an integration test vs a unit test, I care that it tests _REALITY_.
There are times when the isolation given by mocking can be useful, but I submit it's useful far far less often than many claim. And your examples are a good use as well.
But imo trying to mock everything is mostly mental masturbation only with long-term maintenance costs.
https://blog.mockadillo.com/posts/the-importance-of-mocking-...
Poorly named libs (e.g. mockito) just exacerbate this problem.
The way I see it mocks are the all encompassing tool here. With mocks you can both stub behaviors and, if you want, verify behaviors.
And in reality every mock needs to act as a stub, e.g. a mocked method returning a preset value. So, it seems natural that "mocking" as more encompassing concept is what is used in library names. If people end up using those to mostly create stubs instead of mocks so what?
My experience so far has been that people naturally flock to a solution for testing that is flexible and lets them quickly cover the boundaries of the code they are testing. If people want to check if their code calls a method x times but are not interested in what that method does for the test, they use mocks. If they do they use "the real thing" or a fake on which they can validate state later. And if they just need a method to return something and don't care about whether it's called or how often they use a stub.
I feel that often when you stub you can also go ahead and add code to verify some behaviors (mock) to get some extra coverage. I don't know I just don't think it's bad to mix and match if it gets you to more things tested.
Nothing wrong with mocks of course, sometimes they are absolutely required. I just favor isolation using stubs, and using mocks sparingly and only when absolutely needed.
> ...in reality every mock needs to act as a stub I disagree. In fact a collaborator method call that doesn't return a value is a great candidate for mocking. Conversely, a collaborator that does return a value can usually be stubbed.
> ...when you stub you can also go ahead and add code to verify some behaviors (mock) to get some extra coverage Entirely dependent on context of course, but to generalize, this feels risky. In many cases adding extra coverage via mocks might be making tests more brittle, for little to no value gain. Further if every method you mock is returning a preset value, there's no reason to verify expectations on it, because it's a stub that will allow verifying expectations on the output of the object/function under test.
As far as library naming, it's just a pet peeve stemming from "mock" somehow becoming synonymous with "test double". If people use a "mocking" library for stubs... well... that's good! :)
I have often - like on a team of 5-10 at least 2x per year - seen people wire a mock to a mock or a fake to a mock, and either not realize it (because it's under six layers of DI) or just they're not paying attention but someone said they "have to write tests" and trying to get it over with. And this is generally a lot less obvious, because the test suite is still full of assertions, sometimes even totally reasonable-looking ones. (Such tests are even sometimes "real tests" - but that the mock is doing what's expected, not of the desired SUT.)
There's also the general argument from the principle of least power. The less code in your doubles, the more certain you are the doubles are doing what you expect, and the more you're verifying what really matters in your code. Like how pure functions are preferable to impure functions when possible, having comprehensive tests without concerns about "side effects" also means your code has fewer relevant side effects.
The example outlined in the article isn't exactly a unit test: The change in state of one object changes the state of another object. If these are coupled that tightly I'd rather try to mock (stub) the dependencies of both objects and let them interact (test both at the same time), then verifying their final state.
In essence classic xunit tests are not just unit tests, but also mini-integration tests. As a result many people like the fact that client tests may catch errors that the main tests for an object may have missed, particularly probing areas where classes interact. Mockist tests lose that quality. In addition you also run the risk that expectations on mockist tests can be incorrect, resulting in unit tests that run green but mask inherent errors.