This is true only if you adhere to the common definition of "unit test" where a unit test must test a single class or method on a class, often with use of lots of mocks.
Common but not necessarily best.
Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well?
Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit".
"unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1)
I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level.
Well, Agile also used to make sense before it was called Agile and turned into a set of rituals.
Same for unit tests and any other practical matters that have turned into religion.
Plus there's the little matter that you tend to think that your application is tested if it has automated tests. Then your colleague in sales who clicks like he's having an epileptic episode comes over and your app crashes 3 different times [1].
[1] True story. I used to ask him to click 2 min on whatever I was working on when he passed by my desk.
[2] PalmOS had the gremlins tool. To my knowledge no other platform has it.
> Well, Agile also used to make sense before it was called Agile and turned into a set of rituals.
If you want to make sense of "Agile" read "The Agile manifesto" (1), it's not long even with the 2nd page (2).
but yes, the modern criticisms of modern "agile" are pretty much the same as the problems that caused that agile movement in the first place. And are starkly at odds with the Agile Manifesto, e.g. "Individuals and interactions over processes and tools" and "Responding to change over following a plan" vs "Our tool, Jira, enables micromanagement of human resources over a long-term plan"
But that value is not in yet again, before each release, taking a day to run through an excel spreadsheet of checks that should have been long ago automated into a 2-minute run.
But some tests will take time to run so a day turn round is a quick turnround.
If you have databases involved it will not be a 2 minute run - the network latency will make the queries take time. Note these need to be run as integration tests so mocking is not sufficient.
2 minutes is an eternity in database time. You should be able to run dozens of test cases in that amount of time unless your app is just monstrously slow.
The time in the database is low. The time is the time transferring data over the network and the limiting factor is the number of individual transactions. A result processing one record does have similar speed to one looking at 1000 records.
The concept of refactor does not apply in context of publicly exposed interfaces—i.e., you logically never “refactor” a public API, you release a new version of it. As such, it would have different inputs and outputs and therefore its own separate tests.
This becomes more difficult to reason about when what you build is not just a library but something more like a web service or a GUI/CLI app. For a library, a unit test may be able to cover public API and be an end-to-end test in effect, whereas for a more complex product intended to exist over time in production unit tests do in fact provide less of a guarantee, and versioning public API becomes difficult (unless you immediately drop support for any non-current API version).
It is useful to be able to refactor the code behind that interface, or 1 or more levels down the call stack, while testing via that public interface.
Contract changes is one reason for such a code change, but there are many others - including bug fixes, error handling, handling input cases that were not well supported before, performance or readability improvement, changing to use a new version of an internal library with a new contract, and pure refactoring to e.g. extract common code, with no other logical change.
it is useful to do such incremental changes supported by tests on a "publicly exposed interface" that does not change frequently, instead of tests coupled to classes that are not not part of a "publicly exposed interface".
Absolutely. If there is no logical change to a public interface, then your e2e tests (irrespective of whether they are unit tests, where units are strictly public members or classes or modules, or some other kind of tests) should not change either.
> For as much as companies have focused on DevOps and continuous delivery, the sad truth is that the trend of releasing software has remained relatively flat, with only 10% of teams releasing at least daily.
Sad truth? Only? How about an alternative interpretation: 90% of teams don’t need or want to release at least daily. And so they don’t.
Continuous delivery is something that 90% of software teams don’t need and shouldn’t waste time on. It’s been overhyped.
Releasing code to prod is the end of the feedback loop, until then you have possibly working software. The long you wait until release, the less likely it will work as you expect.
You don’t need to release to users, that can happen later.
Every team I’ve been on that could release daily never wanted to go back to anything less.
The tooling is still too hard for mass adoption by teams, but the benefits are real.
You’re both conflating two issues. A solid fully automated CI/CD is table stakes for any professional project. Releasing on some particular cadence has little to do with this aside from the fact that it isn’t possible without such automation. In many contexts I’d consider security and audit ability far more important than release pace, and even on personal projects you want the deployment to be reproducible and explicitly defined.
I disagree about continuous delivery. Small, regular releases mean less risk in each release. Basically every company I've worked for has used CI/CD, although 'release daily' seems arbitrary (you release when you merge - that's the whole idea. If there's nothing merged today, we have nothing to release).
Most of the code I have written in the last years has been request response centered. For these I find integration tests to be the best testing strategy. I'll add the occasional unit test for logic heavy operations.
I think the real value in unit tests isn't validating your logic is accurate (although sometimes it is for logic heavy methods as you say), but in validating that subsequent changes (often made by other devs) don't break prior logic.
In complex codebases it's easy to miss things when making changes and unit tests are great ways to catch issues in really simple bits of logic that have just later been overlooked.
Integration tests are typically so heavy and multivariable they can only be used to validate the most common scenarios. They're great sanity checks, but provide terrible coverage whenever there exists nuanced logic.
This is the first and only article by this guy on that web site. He sells browser automation.
I tried finding non-platitude insights but couldn't.
His conclusion is:
> By embracing a more holistic testing approach, teams can deliver better software faster. As with so much in life, the key is using the right tool for the job, not just the tool that is most convenient at the moment. When it comes to testing, smart and strategic beats brute force every time.
Note how this statement conveys zero information and could be said about literally anything.
If you're looking for an article with substance and a similar bent (functional specifications driving decisions on test granularity), I'd try this one:
My favorite hot take from that article is that if a functional test can't exercise a piece of code, delete it. That is, if the code can't be exercised in normal usage of your program, it serves no purpose.
One retort might be that sometimes code defends against malicious actors who, say, have direct access to private endpoints etc., but if you're defending against such threats, either that requirement drives your functional spec (e.g. banking infrastructure) or maybe you're taking things a little too far.
No they're not. Unit tests are only overrated when you're hacking some shit together rather than doing forward engineering. No one does proper forward engineering these days which is why it doesn't seem applicable. If you have thought about the problem first you know the constraints and they match 1:1 with the test cases, then you build it, then it works. You can't test or understand side effects for everything at the boundary of a system from experience.
The biggest sin of the whole IT sector at the moment is lack of quality and this thinking drives it further towards hell.
This POV is debateable (but I sympathize strongly with your sentiment). If quick but shitty allows me to capture a lot of value then quick and shitty might be the way to go. Yes there are major downsides but especially in a "scaleup" setting they are often dwarfed by the upsides of shitty but fast. As an engineer, I hate it because I want to build perfect and long term all the time but in many cases you can't really argue with the monetary benefits of knowingly delivering underwhelming things that get the job done well enough for now (even knoing deep in your heart that many of the issues might never be fixed).
A good example is security. Sometimes it's righ to ship stuff with security issues that you know exist and might even be 10+ million dollar issues if discovered quickly if for example fixing thoses issues might interfere with adoption because they make user onboarding slightly more annoying.
Whilst the core message of your comment I agree with, unit tests are not the only way to achieve this; but there's often blanket statements made about 100% code coverage, and all that nonsense - which really just fetishizes unit testing rather than critically thinking about the goal of a unit-test.
For example, better use of types to make sure applications physically can't get into a bad state will always have better returns in terms of velocity and correctness than checking after the fact with unit-tests. It has the additional benefit of being compositional, unlike unit tests.
This can be done for data representation but also transitions of state, which is all of software engineering outside of IO.
Ironically the goals of unit tests are almost always better served by higher level integration tests - tests that actually call APIs, put data in the database and retrieve it, etc.
> removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing
That's the worst idea ever. Testing requires completely distinct set of skills different than development. There are 10x programmers but there are also 10x testers.
Yeah there's some poor thinking by the original article.
Developers must be accountable for their code quality yes but must be held accountable by an external force. I've been in too many organisations where empty, useless unit tests are generated to build coverage and say "look we are accountable" when they are not really. That might be an external QA / review team under a completely different management tree to the development team.
Realistically I've never seen an org where it's possible to scale it up and self-accountability being viable. There are too many defective personalities to manage this.
He's not promoting that removing dedicated testers is a good idea. Full(er) quote:
> Several strategies are getting deployed — shift left (e.g., more unit tests), shift right (e.g., test in production), transitioning from manual to automated testing and even removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing.
>
> These testing strategies are met with varying degrees of success and can even differ from team to team based on skill level, maturity, and culture.
By all means, have developers write their own unit tests. To show their intent, document use cases, etc. And do not leave that to a junior colleague or a brain dead AI, optimizing for high McCabe complexity code coverage or whatever metric.
AND (indeed) have QA done by someone with different skills. Development requires a focused mindset, in the zone for long stretches, int the box. Test requires distraction, exploring, out of the box — "hey, that's weird" or "I wonder what will happen if I do this".
Trying to avoid shipping bugs is a defense-in-depth affair, because fundamentally tests can only show the existence of a bug, but not that a program is bug-free. It starts right in the editor, where syntax highlighting can help with spotting mistakes such as writing 010 instead of 10 (0 prefix for octal is a terrible idea).
Pretty sure that bug-free code is one of the prides of programmers worth their salt; but there's no helping, when you forget about something while coding, it is very likely that you also forget to test it. That's why we need a separate tester.
> TDD itself is also overrated, as writing testable code is much more important than the actual tests.
Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
> This is why getting artificial intelligence projects to create basic tests isn’t the impressive accomplishment people seem to think it is.
Also correct. Additionally, the point of unit tests is to document, to show the code's use case. It's generally not possible to device intent (why) from implementation (how), but if the AI were magically able to deduce that that's a sign that the code is either trivial, or plagiarized (or both).
> By embracing a more holistic testing approach, teams can deliver better software faster.
Sure, if velocity is your objective. Another approach is to stop and think about whether you're testing in internal (unit test) or external (integration, acceptance test) quality. As a developer, I use unit tests primarily to verify complicated business logic to prevent the embarrassment of obvious flaws in my code further downstream.
> > TDD itself is also overrated, as writing testable code is much more important than the actual tests.
> Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Also, your code can work, with 100% unit test coverage, and have the tests test nothing useful. So reasoning about the quality of your code through the quantity of your unit tests is highly suspicious.
And btw, TDD does not advocate code quality and well organized code - if anything, it lends itself to the opposite.
> You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful.
Totally true. But give me those unit tests, and I can tell that the code is sh*t.
If I have trouble understanding the source, it could be because it's doing complicated stuff. It could be highly optimized. It could I'm too dumb.
If I have trouble understanding the unit tests demonstrating what the code does, there is either something wrong with the tests or the implementation. Fix it, rinse repeat.
I wasn't saying that unit tests or code coverage are a panacea or that TDD guarantees great code. But it's a tool. It's better than writing documentation, but it serves more or less the same purpose. Sometimes it's useful, sometimes not. And its complete absence is usually a bad smell.
Types are wonderful, and a good type system makes so many errors disappear / impossible, that I'd otherwise have to write tests to cover in a dynamic language.
Types however, are not a substitute for unit tests. No type system is going to tell me if my income tax calculator covers all the edge cases.
I would argue that your income tax calculator logic should probably not be unit tested, it sounds complicated enough to involve multiple classes / units of code. What you really want to test is business logic, not code logic.
I was using the example of income tax as an entirely theoretical example of where types won't save you in response to the GP.
That said, I don't follow what you mean by "to test is business logic, not code logic." To me, that is a distinction without difference.
An income tax calculator in a (sane) tax system is essentially a set of pure of functions or classes. This is a perfect place for a unit test. I might have additional unit tests for sub components of the hairy bits, but it's still unit tests all the way.
Maybe _your_ type system doesn't do that. It is perfectly possible to introduce type aliases to differentiate between X and Y coordinates - though usually it is not worth the trouble. (Unless you are working on the firmware of a nuclear missile - in which case it probably is :))
I personally have a lot of informations coming from my text editor (emacs) when writing code, and I only want to say that your example does the opposite of what you meant; because of the type system, typing Rectangle TAB would likely print Rectangle(width, height) automatically and inform you with "width, a 32 bit float; height, a 32 bit float".
I fail to find a better example (complex functions of high arity with parameters from the same type ?).
It depends on what language you are using and what things you are testing.
If you use a language like Python, having tests that pass things that aren't the type you are expecting are useful. Likewise passing nulls to a nullable parameter, or different objects to an Object/Any type (or even a different implementation/type to a supertype parameter).
Those tests are part of the unit test repertoire, but unit tests should cover both expected cases/behaviour and unexpected cases.
Sigh. Releasing more often is not an ideal situation. Yes, being able to release every single change on its own, fully tested, is a nice to have tool. But the real world works in office days, weeks, and quarters. And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass.
Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests.
But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
>And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass.
I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down.
>But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part.
The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly.
Some bugs/issues appear over longer time spans. Things like issues around daylight savings time. Or memory leaks (or file size limits) that build up over a long time. Or even just accruing a lot of data in a system to the point where it starts exhibiting performance issues in certain workloads.
Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a good dev should be able to do but "wont have time for" but a QA on 40-80% of their salary won't ever manage.
The weird thing is that this really isnt a technical problem. It's an issue with social class.
It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand.
IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations.
Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs.
It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day.
A longer test/release cycle would have made this problem far worse not better.
It didn't work exactly right of course because the docs weren't exactly right.
Indeed. That approach also won't pick up changes to the remote API that break your integration.
An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox.
This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.
To be able to test them, you need to be able enumerate them. Part of the problem here is that you don't always know what regularly occurring external thing is going to cause the problem.
This idyllic view sounds theoretical to me. What I've seen working best is a fast CI/CD pipeline with unit tests, key journey tests, and integration tests with blue/green deployments. This in my experience is far superior to staging environments for both startup and very large scales. In addition the development environment itself should be as close to prod as is practical so most checks can be done there.
Most criticisms of unit tests in articles like this are actually criticisms of bad unit tests.
They also overlook a large part of the value of unit tests, which is fast feedback during development time.
We could push all of our tests up to the browser level, but then I lose the almost instant feedback that unit tests give me.
Sure, if your tests mock everything, and only test individual methods, then they probably are brittle and provide little value over more 'expensive' (e.g. slower) tests higher up the test pyramid.
If the tests only interact with the public APIs (e.g. the 'ports' in a ports and adapters architecture), then they are less coupled to implementation, can represent real business scenarios (instead of 'input x gives result y'), and add more value.
I think I'm disagreeing with the definition implied by the article.
To me, an integration test involves integrating with another system.
Testing groups of functions or classes together is still a unit test, by Wikipedia's definition at least
"unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use" [1]
and by Michael Feather's definition
"A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can't run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing config files) to run it." [2]
Indeed, it's the inevitable result of doing TDD; the refactor step is likely to break out smaller classes/functions already covered by the existing tests.
If we write new, more granular tests every time we do the refactor step, then we end up with brittle tests.
Edit: To clarify, I'm not talking about calling the public API over the network, I'm talking about the public methods that consumers (including your network layer) would call.
To me, an integration test involves integrating with another system.
That's not what it means, and the author clearly indicated it didn't?
When your colleagues are talking to you, you're going to be constantly misunderstanding them if you just make up your own definitions, even after they explained them!
When I was 14 I believed @ meant about and now I no longer use @ to mean about.
Actually no, the "unit" in unit tests is about isolation: the test can be executed as an independent unit, not about a unit of code (single class/method). If you search around, the leading experts who popularized unit tests described it as such.
It is perfectly fine to test multiple classes as a unit, in fact you probably already do so, as I would be extremely surprised if you mock your string class in all your unit tests.
If you have to 'search around' for your definition and make a dubious appeal to authority without citing any of those experts, perhaps you need to wonder if it's your definition that's wrong.
Further in the article he goes over exactly why he feels that its allowed to both test a single class or multiple at once, going as far to give a name to them: solitary vs sociable unit tests.
Then, he goes on to, and let me emphasize this, defend against the criticism that other people make that this type of test is not a unit test. (sound familiar?):
Indeed using sociable unit tests was one of the reasons we were criticized for our use of the term "unit testing". I think that the term "unit testing" is appropriate because these tests are tests of the behavior of a single unit. We write the tests assuming everything other than that unit is working correctly.
I'm not sure how thoroughly you're looking to be refuted here but I feel I can't quite do a better job.
Funny thing is, this is not even a very deep insight: the answer to nearly any question in software design can be boiled down to: it depends. This discussion is just the unit-test rendition of "it depends". Why are you so hell bent on having it exactly one way?
I guess it depends on the software that you are writing and how it's structured. The testing pyramid seems to make some sense, as long as you realize that you need a multitude of different tests for checking various things.
For example, when writing a library that dealt with JWT and called into some other libraries, I opted for a lot of unit tests, I think the code coverage was around 90-95%. Writing tests for almost all of the functionality helped me actually discover one or two bugs which I could immediately fix without having them be released. Also, tests acted as documentation there, showing various use cases for the library, in addition to helping me actually make the API more user friendly.
Integration and UI tests are also useful to see how the solution fares in the real world, but once you have to automate a browser or deal with mocking the DB (if you can't spin up a container, migrate the schema and seed it with real data for your tests), the complexity skyrockets. You might still want to have them and plenty of them, but at the same time you'll deal with slow and flaky tests more often, especially if the tech you choose or are stuck with won't support your needs all that well (e.g. Selenium). It's even worse if the thing you're testing changes a lot.
As long as you have a staging environment, you should be even well off for anything from load testing to even the occasional bit of manual/acceptance testing, or trying to reproduce issues there. With the help of logs/metrics you can reason about changes/bugs/reproduction scenarios more safely, without impact on prod. Even better with other supporting things, like feature flags. Test whatever you can: sometimes the limiting factor will be time, other times the complexity of the system (or how testable it is in the first place, e.g. no fixed element IDs in a web browser), other times you won't get the infrastructure, but it's still a nice pursuit.
72 comments
[ 3.5 ms ] story [ 132 ms ] threadCommon but not necessarily best.
Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well?
Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit".
"unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1)
I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level.
1) https://www.artima.com/weblogs/viewpost.jsp?thread=126923
Same for unit tests and any other practical matters that have turned into religion.
Plus there's the little matter that you tend to think that your application is tested if it has automated tests. Then your colleague in sales who clicks like he's having an epileptic episode comes over and your app crashes 3 different times [1].
[1] True story. I used to ask him to click 2 min on whatever I was working on when he passed by my desk.
[2] PalmOS had the gremlins tool. To my knowledge no other platform has it.
If you want to make sense of "Agile" read "The Agile manifesto" (1), it's not long even with the 2nd page (2).
but yes, the modern criticisms of modern "agile" are pretty much the same as the problems that caused that agile movement in the first place. And are starkly at odds with the Agile Manifesto, e.g. "Individuals and interactions over processes and tools" and "Responding to change over following a plan" vs "Our tool, Jira, enables micromanagement of human resources over a long-term plan"
1) https://agilemanifesto.org/
2) https://agilemanifesto.org/principles.html
Yeah, too bad modern Agile doesn't have much to do with the original manifesto...
Hit multiple keys on the keyborard at once and do that for a few seconds.
Then if it p[assed that he would get down to more structured testing.
Testers have to assume the user is an idiot and also will always do the wrong thing.
But that value is not in yet again, before each release, taking a day to run through an excel spreadsheet of checks that should have been long ago automated into a 2-minute run.
But some tests will take time to run so a day turn round is a quick turnround.
If you have databases involved it will not be a 2 minute run - the network latency will make the queries take time. Note these need to be run as integration tests so mocking is not sufficient.
This becomes more difficult to reason about when what you build is not just a library but something more like a web service or a GUI/CLI app. For a library, a unit test may be able to cover public API and be an end-to-end test in effect, whereas for a more complex product intended to exist over time in production unit tests do in fact provide less of a guarantee, and versioning public API becomes difficult (unless you immediately drop support for any non-current API version).
Contract changes is one reason for such a code change, but there are many others - including bug fixes, error handling, handling input cases that were not well supported before, performance or readability improvement, changing to use a new version of an internal library with a new contract, and pure refactoring to e.g. extract common code, with no other logical change.
it is useful to do such incremental changes supported by tests on a "publicly exposed interface" that does not change frequently, instead of tests coupled to classes that are not not part of a "publicly exposed interface".
Sad truth? Only? How about an alternative interpretation: 90% of teams don’t need or want to release at least daily. And so they don’t.
Continuous delivery is something that 90% of software teams don’t need and shouldn’t waste time on. It’s been overhyped.
You don’t need to release to users, that can happen later.
Every team I’ve been on that could release daily never wanted to go back to anything less.
The tooling is still too hard for mass adoption by teams, but the benefits are real.
In complex codebases it's easy to miss things when making changes and unit tests are great ways to catch issues in really simple bits of logic that have just later been overlooked.
Integration tests are typically so heavy and multivariable they can only be used to validate the most common scenarios. They're great sanity checks, but provide terrible coverage whenever there exists nuanced logic.
I tried finding non-platitude insights but couldn't.
His conclusion is: > By embracing a more holistic testing approach, teams can deliver better software faster. As with so much in life, the key is using the right tool for the job, not just the tool that is most convenient at the moment. When it comes to testing, smart and strategic beats brute force every time.
Note how this statement conveys zero information and could be said about literally anything.
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
One retort might be that sometimes code defends against malicious actors who, say, have direct access to private endpoints etc., but if you're defending against such threats, either that requirement drives your functional spec (e.g. banking infrastructure) or maybe you're taking things a little too far.
The biggest sin of the whole IT sector at the moment is lack of quality and this thinking drives it further towards hell.
Do less, do it slower but do a better job please.
A good example is security. Sometimes it's righ to ship stuff with security issues that you know exist and might even be 10+ million dollar issues if discovered quickly if for example fixing thoses issues might interfere with adoption because they make user onboarding slightly more annoying.
For example, better use of types to make sure applications physically can't get into a bad state will always have better returns in terms of velocity and correctness than checking after the fact with unit-tests. It has the additional benefit of being compositional, unlike unit tests.
This can be done for data representation but also transitions of state, which is all of software engineering outside of IO.
That's the worst idea ever. Testing requires completely distinct set of skills different than development. There are 10x programmers but there are also 10x testers.
Developers must be accountable for their code quality yes but must be held accountable by an external force. I've been in too many organisations where empty, useless unit tests are generated to build coverage and say "look we are accountable" when they are not really. That might be an external QA / review team under a completely different management tree to the development team.
Realistically I've never seen an org where it's possible to scale it up and self-accountability being viable. There are too many defective personalities to manage this.
AND (indeed) have QA done by someone with different skills. Development requires a focused mindset, in the zone for long stretches, int the box. Test requires distraction, exploring, out of the box — "hey, that's weird" or "I wonder what will happen if I do this".
Pretty sure that bug-free code is one of the prides of programmers worth their salt; but there's no helping, when you forget about something while coding, it is very likely that you also forget to test it. That's why we need a separate tester.
Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
> This is why getting artificial intelligence projects to create basic tests isn’t the impressive accomplishment people seem to think it is.
Also correct. Additionally, the point of unit tests is to document, to show the code's use case. It's generally not possible to device intent (why) from implementation (how), but if the AI were magically able to deduce that that's a sign that the code is either trivial, or plagiarized (or both).
> By embracing a more holistic testing approach, teams can deliver better software faster.
Sure, if velocity is your objective. Another approach is to stop and think about whether you're testing in internal (unit test) or external (integration, acceptance test) quality. As a developer, I use unit tests primarily to verify complicated business logic to prevent the embarrassment of obvious flaws in my code further downstream.
> Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Also, your code can work, with 100% unit test coverage, and have the tests test nothing useful. So reasoning about the quality of your code through the quantity of your unit tests is highly suspicious.
And btw, TDD does not advocate code quality and well organized code - if anything, it lends itself to the opposite.
Totally true. But give me those unit tests, and I can tell that the code is sh*t.
If I have trouble understanding the source, it could be because it's doing complicated stuff. It could be highly optimized. It could I'm too dumb.
If I have trouble understanding the unit tests demonstrating what the code does, there is either something wrong with the tests or the implementation. Fix it, rinse repeat.
I wasn't saying that unit tests or code coverage are a panacea or that TDD guarantees great code. But it's a tool. It's better than writing documentation, but it serves more or less the same purpose. Sometimes it's useful, sometimes not. And its complete absence is usually a bad smell.
I think Michael Feather's book demonstrates how code improves by simply making it testable https://www.oreilly.com/library/view/working-effectively-wit...
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
Types however, are not a substitute for unit tests. No type system is going to tell me if my income tax calculator covers all the edge cases.
All tests should be on business logic - code logic might not fit the business case at all and so that should be a fail.
That said, I don't follow what you mean by "to test is business logic, not code logic." To me, that is a distinction without difference.
An income tax calculator in a (sane) tax system is essentially a set of pure of functions or classes. This is a perfect place for a unit test. I might have additional unit tests for sub components of the hairy bits, but it's still unit tests all the way.
I fail to find a better example (complex functions of high arity with parameters from the same type ?).
If you use a language like Python, having tests that pass things that aren't the type you are expecting are useful. Likewise passing nulls to a nullable parameter, or different objects to an Object/Any type (or even a different implementation/type to a supertype parameter).
Those tests are part of the unit test repertoire, but unit tests should cover both expected cases/behaviour and unexpected cases.
Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests.
But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests.
Is anyone working on a reinforcement learning agent for proper monkey testing? :)
I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down.
>But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part.
The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly.
The actual solution is in my username.
The weird thing is that this really isnt a technical problem. It's an issue with social class.
IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations.
Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day.
A longer test/release cycle would have made this problem far worse not better.
Indeed. That approach also won't pick up changes to the remote API that break your integration.
An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox.
This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.
They also overlook a large part of the value of unit tests, which is fast feedback during development time. We could push all of our tests up to the browser level, but then I lose the almost instant feedback that unit tests give me.
Sure, if your tests mock everything, and only test individual methods, then they probably are brittle and provide little value over more 'expensive' (e.g. slower) tests higher up the test pyramid.
If the tests only interact with the public APIs (e.g. the 'ports' in a ports and adapters architecture), then they are less coupled to implementation, can represent real business scenarios (instead of 'input x gives result y'), and add more value.
You've fallen into the very trap of terminology the author mentions at the beginning of the article.
Testing groups of functions or classes together is still a unit test, by Wikipedia's definition at least
"unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use" [1]
and by Michael Feather's definition
"A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can't run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing config files) to run it." [2]
Indeed, it's the inevitable result of doing TDD; the refactor step is likely to break out smaller classes/functions already covered by the existing tests. If we write new, more granular tests every time we do the refactor step, then we end up with brittle tests.
[1] https://en.wikipedia.org/wiki/Unit_testing
[2] https://www.artima.com/weblogs/viewpost.jsp?thread=126923
Edit: To clarify, I'm not talking about calling the public API over the network, I'm talking about the public methods that consumers (including your network layer) would call.
That might be where the confusion lies here.
That's not what it means, and the author clearly indicated it didn't?
When your colleagues are talking to you, you're going to be constantly misunderstanding them if you just make up your own definitions, even after they explained them!
When I was 14 I believed @ meant about and now I no longer use @ to mean about.
It is perfectly fine to test multiple classes as a unit, in fact you probably already do so, as I would be extremely surprised if you mock your string class in all your unit tests.
https://martinfowler.com/bliki/UnitTest.html
Martin Fowler
Although I start with the notion of the unit being a class, I often take a bunch of closely related classes and treat them as a single unit.
https://medium.com/@_ericelliott/i-use-the-well-known-defini...
Kent Beck
Unit tests test individual units (modules, functions, classes) in isolation from the rest of the program
Such great evidence.
Then, he goes on to, and let me emphasize this, defend against the criticism that other people make that this type of test is not a unit test. (sound familiar?):
Indeed using sociable unit tests was one of the reasons we were criticized for our use of the term "unit testing". I think that the term "unit testing" is appropriate because these tests are tests of the behavior of a single unit. We write the tests assuming everything other than that unit is working correctly.
I'm not sure how thoroughly you're looking to be refuted here but I feel I can't quite do a better job.
Funny thing is, this is not even a very deep insight: the answer to nearly any question in software design can be boiled down to: it depends. This discussion is just the unit-test rendition of "it depends". Why are you so hell bent on having it exactly one way?
For example, when writing a library that dealt with JWT and called into some other libraries, I opted for a lot of unit tests, I think the code coverage was around 90-95%. Writing tests for almost all of the functionality helped me actually discover one or two bugs which I could immediately fix without having them be released. Also, tests acted as documentation there, showing various use cases for the library, in addition to helping me actually make the API more user friendly.
Integration and UI tests are also useful to see how the solution fares in the real world, but once you have to automate a browser or deal with mocking the DB (if you can't spin up a container, migrate the schema and seed it with real data for your tests), the complexity skyrockets. You might still want to have them and plenty of them, but at the same time you'll deal with slow and flaky tests more often, especially if the tech you choose or are stuck with won't support your needs all that well (e.g. Selenium). It's even worse if the thing you're testing changes a lot.
As long as you have a staging environment, you should be even well off for anything from load testing to even the occasional bit of manual/acceptance testing, or trying to reproduce issues there. With the help of logs/metrics you can reason about changes/bugs/reproduction scenarios more safely, without impact on prod. Even better with other supporting things, like feature flags. Test whatever you can: sometimes the limiting factor will be time, other times the complexity of the system (or how testable it is in the first place, e.g. no fixed element IDs in a web browser), other times you won't get the infrastructure, but it's still a nice pursuit.