I don't get it. In their example of fibo(15), they already wrote the fibo function but how do they know it is correct ? Ok it is nice to have an autofill for expected value but not from the same 'thing' that I want to test...
I'm not the OP but I think Fibonacci is a contrived example. In practice these "characterisation tests" are great for adding tests to an existing codebase. The "snapshot" just records the current behaviour; it doesn't judge whether it's correct or not, but if you refactor it will catch changes to the behaviour, and then it's up to a human to judge if the changes were desirable or not.
I am always slightly uncomfortable about property based tests that just replicate the logic of the thing they're testing. I don't really know of an alternative in this case, and obviously there's rarely any harm in having regression tests. But it feels like a code smell somehow.
I suppose they're ideal for cases where there's a simple, obviously correct but slow implementation that you can use to test a more complex but faster implementation.
You say it replicates the logic, but I think it's more useful to say that it describes the meaning: fib is the function on natural numbers that has that property plus the base cases fib(0) = 1 and fib(1) = 1. That's what fib means.
That said, there are plenty of cases where your reaction is justified, where the test is implemented the same way as the function it's testing when the correctness property could be specified more simply.
Your uneasiness is understandable, but often it is far simpler to check the correctness of a result than to find it in the first place. Then property-based testing really shines.
Even outside that special case it is always possible to generate just some input data and check for the absence of runtime errors or exceptions. That approach can be improved by adding assertions, pre- and postconditions to your code under test.
I think what they've done is implemented a way of doing snapshot testing by putting the snapshot data in the test itself. In snapshot testing you just cache the result of a code block and use that cached data to check the code produces the same output as it did when you knew it worked. Putting the snapshot inline in the test makes it a lot easier to read.
Maintaining tests that only test that things haven't changed can be a nightmare if you actually want things to change (like when you're making a change to the logic of a system). They can also be a real pain to maintain since if they fail you almost never know why they failed, just that they failed. This gives you (or the future maintainer) very little help in figuring out what's going on.
Guidelines for tests that reduce maintenance burden rather than add to it:
* Test the externally visible useful properties of your system, rather than implementation details. Related: don’t bother with mocks unless there’s no other way
* Keep your tests DRY: if there’s some invariant you want to check have it appear once in your test code
* Don’t over-test: you can delete tests and checks that fully overlap with other tests
* If you need to over-test make it easy to regenerate the expected results from your implementation (characterisation tests) as espoused by the article.
* the amount of testing effort should be proportional to how important the functionality is. Reduce the effort for low impact features
That’s a big assumption. In TDD you write the tests first, to help you achieve the results expected, this doesn’t fulfill that need. It is basically snapshot testing applied to any kind of function.
I am of the opinion that the true value of TDD is that tests get written at all. I suspect that in practice the tests tend to get written at roughly the same time as the code, rather than strictly before.
IMHO, this is really more a documentation of the original behavior and maybe a test for consistency of later implementations against the original one. But it's not about validity. I'd call it a "traditionality test".
Not fib, but... I've written tests like this semi-recently for an algorithm that I had to devise. I went to Excel to generate the test data, based on a data pull from our production database. So, real production data, using a sequence of Excel spreadsheet functions, which also has multiple eyes on it (including legal, since this was regulatory in nature), fed as input to the algorithm to verify the output matched up.
I'm still on the fence about that but I'm not sure how else to test something that is essentially a glorified algebra formula.
Fix your footer links. I was trying to get to your main page to find out who you are (already a fail if I have to use your footer over the Jane Street logo at the top) and five of your links 404, including the first one I tried. Not a good look.
That's fair criticism but I can assure you that Jane Street don't need to convince anyone about anything. If you don't already know who they are and what they do, your opinion doesn't matter to them. They wouldn't even need to have a functional website for all that matters.
I was genuinely expecting another article about ChatGPT and some ingenious way to use it for generating test-cases.
Turns out, this is just marketing post after all with no relation to the former... Disappointing...
Humans are better at verifying the answer is correct over coming up with the answer.
The thing that makes test writing unjoyous for me is most code is hard to test, usually requiring all kinds of scaffolding and even resources like files, db, or mocked http requests.
Being Jane Street they probably use OCaml so yes functional programming languages are nicer to write tests
in.
Ah neat, I've heard different names for snapshot/golden master type testing before, but generally find the tooling a bit lacking. It seems like https://github.com/approvals/ actually has pretty decent implementations for most languages.
I feel like the Jane Street approach of embedding the snapshots directly in test sources would make a qualitative difference to the ergonomics. The post implies that this requires some emacs integration, but there's no reason it couldn't be done by a test runner process instead. It would also be cool to have an LSP that provide a hover text or go-to-definition for externally stored snapshots.
> usually requiring all kinds of scaffolding and even resources like files, db, or mocked http requests
Yes, anything with timestamps or generated identifiers gets very annoying as you have to manually exclude certain data from the snapshots, negating some of the "easy wins" a snapshot/approval process provides for pure/idempotent code.
A VSCode plugin could inline the expected answer file, or at least link to it. I like a seperate file because what if you want whitespace checkin behaviour to be different?
I am not an software engineer, but I write code almost daily for personal side projects and automation at work. I write unit and integration tests even for the most toy thing I do (this doesn't come from some kind of fanatic view about testing, but simply because I enjoy writting code and learning/exposing myself to the reasoning and decisions of certain scenarios). I understand the benefits of it, however, I find unit testing specifically tedious and boring as hell. I always wondered if it could be done by some framework that relies on something like Github Copilot: Install the framework, run it pointing to your function/methods file(s), get a file with your tests that you can perhaps adjust here and there and then integrate it in your CI. I understand the challenge due to, perhaps in some cases, the implicit difficulty of coming up with relevant tests for certain parts of your business logic but, do you think it will be possible some day, Am I saying something stupid?
Not at all stupid. I'm sure people are already using ChatGPT to generate unit tests. There are limits, of course, given that (for now) it doesn't have the full context of your code, but it's definitely capable of generating tests for pure functions that are named well and don't require too much outside context. Some projects have tons of these.
code with output that can only be evaluated artistically can't be called right or wrong anyway. As long as it throws no errors, it's a valid language model. I can only imagine how many nulls and undefineds and Infinitys they just throw out and shove under the rug. That's actually the kind of code where not knowing how it works is okay. Encouraged, even.
chatGPT works surprisingly well for python (and am sure other popular languages too). I can dump a code bit and tell it to write the tests + fixtures for me. The tests it wrote actually showed it "understood" (or recognized the pattern of) the code too.
For example part of the code renamed some columns for a data frame that was loaded from a csv. The fixture it created for the csv had the correct column names the code was renaming.
Unless you're doing TDD or chatGPT is too busy, there's almost no excuse now to not write some unit tests ;)
Yes, I use it in that way. And if ChatGPT didn't generated the code with pure functions (usually it didn't), you can explicitly ask to generate the code with pure functions. Then ask to generate the tests.
Usually I get good tests from ChatGPT when I approach it as an iterative process, requesting multiple improvements to the generated test based on what it gives to me. Note that it doesn't replace the skillsets you need to know to write good test coverage.
For example, you can ask to generate integration tests instead of unit tests in case it need context. Providing details on how the testing code should be generated really helps. Also, asking to refactoring the code in preparation to make it testeable, for example, or requesting to convert some functions to actual pure functions, or requesting to refactor a piece of code it generated to a separate function. Then you ask to generate tests for normal and also for boundary conditions. The more specific you get, the chance of getting a good and extensive tests from it is much higher.
Having the tests and code generated by ChatGPT really helps to catch the subtle bugs it usually generates on the generated code (fixing it manually), usually I get test coverage that proof the robustness I needed for production code.
This approach still needs manual fine tuning of the generated code, I think ChatGPT still struggle to get the context right
but in general, when it makes sense to use it, I'm more productive writing tests in this way than manually.
I used my Copilot trial to explore writing tests more than writing code. I found that it actually worked well, especially for data and boilerplate heavy tests, like table-driven tests in Go. It was saving me quite literally dozens, if not a hundred or more, of keystrokes. I wrote Go for side projects these days, not for work, so I don't pay for the license, but if I was writing Go again professionally I'd pay for the license for this alone.
In my day job, I write Ruby, and it didn't impress me much when I used it with Rspec. I'd says it was saving me maybe a dozen or so keystrokes in total to write a new spec.
This may sound weird, but I've written code professionally for 25+ years and I have never written a unit test, I'm not even sure what a "unit test" is or would look like in the context of what I write.
My job is to first consider a dozen ways a new feature might be used; then consider how it could be misused; then write a backend portion and UI for it, testing it for usability and potential for error along the way, using everything I've already thought could go wrong; then monitor it once it's deployed to see if anything actually does go wrong. After actually testing it through everything I can think of by hand, function by function, line by line as I write it, it seems totally stupid to write code to test my own code.
Not to sound like a shit, but that's what end users are for ;)
That's awesome to hear! I myself do something similar with my own code, but then again, I also trust myself more than any computer ;)
About how long would you say that process takes you? Do you repeat it every time a minor change needs to happen to your code? If not, how do you verify the impact of your changes?
Rough answer, I spend about 50% of my time just thinking about pitfalls before I actually write the code, 10% of the time writing it, and the other 40% testing it to my satisfaction (doing every stupid thing I can think of someone doing to break it) before I deploy.
Usually though, in practice, this looks like tracing/logging several times in the middle of each function to make sure I'm getting it right. Where unit tests are supposedly most handy is when unexpected types are passed to functions, leading to unexpected errors. The majority of proofing against that can be done by strict typing and good code documentation.
Ultimately the best test is when you ask yourself "wait, what if someone sent this" and you try it to see if you can break your own code. That's just a spidey sense in the back of your head. If you didn't have the specific doubt to begin with, I don't know how you could write a unit test to disprove it anyway.
[edit] just a note to any new coders reading this: Spend the time to always read, read, read everything about how to harden the code you write for security purposes. Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.
i.e. you're the only one who can update your code because only you know what it's precisely supposed to be doing. Also if you come back in a year you'd better be able to remember everything.
I am sorry if I am being a bit mean - I just have had to take over maintenance of things like this and have found it a nightmare from hell so far.
I think code comments and breaking everything into well-named functions, along with ample side documentation, make it fairly clear to follow. The business logic gets confusing - but anyone who had to work on the code would need to understand that first anyway. I don't think the code itself is difficult. It's all in the docs. Frankly, I don't always remember how something should work and I need to spend a few hours to re-understand some tricky piece of 10 year old code again before I change it. That's the job.
My main concern, of course, is not leaving a mess for myself. That should be every coder's priority. Then taking over someone else's projects wouldn't be so hard.
The problem with anything you've been doing for 25 years is there's not that much chance that I, or anyone else, will convince you to do it differently. That is true of me too of course.
Nevertheless, I highly recommend having automated tests and running them in the presence of memory profilers etc like valgrind if you're using a compiled memory unsafe language like C or C++.
I would look for another job if I had to work with someone that refused to write them to be honest. It's not that I love tests, but I have had to fix other people's bugs.
> Ultimately the best test is when you ask yourself "wait, what if someone sent this" and you try it to see if you can break your own code. That's just a spidey sense in the back of your head. If you didn't have the specific doubt to begin with, I don't know how you could write a unit test to disprove it anyway.
I can't speak for others but that's precisely how and why I write tests. It goes like "Wait, what if someone sent this" -> check what happens -> write a test to automate that check -> optionally, fix the code to handle this specific case.
Then, over time you accumulate tests that check all these weird edge cases for you and it doesn't take much to run them over and over again every time you change the code.
> Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.
I understood both this quote and the one I quoted above as OP arguing that automated testing will not save one from thinking about and fixing possible edge cases therefore automated testing has no value.
I agree with the first part but not the conclusion. My point is that with both manual and automated testing, you still needs to think. It's just that automated tests let you build an executable knowledge base of all the edge cases, errors, security issues etc. you've thought about in the past, and run them with every code change. Hence the value.
That’s great if you’re the only person working on the code.
Not everyone is as diligent and not everyone will have thought through/remembered all those edge cases when they happen to end up maintaining/adding to your code.
Personally I find that tests are a great way to ensure all that hard work you did thinking about those edge cases isn’t wasted. That and manually testing stuff is annoying after the first time.
Things that either work or don't, don't really need tests. Things that have a ton of edge cases... you might spend ten times as long writing tests as writing the code. In a few cases it might be worth it. Like, I have a piece of hotel software that lets rates be set by the night plus peak extra rates over overlapping time periods, and has to delineate the nightly totals for a customer stay that crosses over lots of those. Lots of edge cases. I had that in production for 5 years before someone noticed that although the totals were right, if the very last night was at the default rate after coming off a peak rate, the line item wasn't printed on the bill. It was a simple case of greedily including one more
segment in what was printed. Now... would a unit test have found that? You have to know exactly what you'd be looking for first. The test would have to be smarter than the person writing the code.
No, the test would not help to find it, but now you can write a test for that bug. Later, when you change something, related to this code or something unrelated, you can be sure it did not affect this part.
Tests are useful when you change code that is unrelated, but still somehow connected. You would never notice the issue as you don't manually try the unrelated code, but the test would catch it.
If you had unit/integration tests for all potential edgecases you came up with, you don't have to manually test that over and over.
yeah but because the code is segmented into very discrete functions... and it's quite easy to see their call chains...
Well, I get it. You're right in principle. The truth is, though, I have had to rewrite software from scratch in new languages every 10 years, and these sorts of bugs only appear a few times per decade. Once a piece of central business logic works, it usually works forever; a change to that central logic will of course require changes and tests across the whole system, from user inputs to annual reports, but that can't be helped. I suspect you'd then have to rewrite the unit tests, too.
It's probably true that when I die, most of my software will slowly wind down and eventually be abandoned. But also that's probably true even if I were to write a lot of tests.
> Now... would a unit test have found that? You have to know exactly what you'd be looking for first. The test would have to be smarter than the person writing the code.
This is a textbook example of something that property based testing can catch. Yes, the tests are smarter than the person that wrote the code. At least, smarter in a different way or maybe it’s that the people who wrote the property based testing framework are super smart.
Look up QuickCheck for the OG or property based testing + your language.
I don’t exactly understand your bug, but the way it would work is something like you’d make some generators that create night stays and rates and put those together into a bill. Then you test the property that the number of line items on the bill is the number of nights + number of discounts (or whatever is right). It will search to break this property and if it breaks it, find a minimal case. Seriously check it out, it’s perfect for this sort of thing.
> Not everyone is as diligent and not everyone will have thought through/remembered all those edge cases when they happen to end up maintaining/adding to your code.
Edge cases could be due to accidental complexity. In this case, the solution is to remove accidental complexity, not to freeze it forever in tests.
Edge cases could be due to essential complexity. In this case, if people are making changes without remembering them, it means they are changing code they don't understand. Sure, tests are making it easier. I don't want it to be easier.
> tests are a great way to ensure all that hard work you did thinking about those edge cases isn’t wasted
It is also a great way to ensure that your best talent is wasted on building guard rails.
Think about it: your best engineers are spending time making worst engineers more productive. And they are not even doing it through mentoring, so your worst engineers will remain where they are.
I think if you're working alone or on a small team on a small project it's use is negligible. The added tidium may not be worth it, and many major issues can be shaken out with integration tests, QA and users.
I think a lot of people miss the (in my opinion) best use case for UnitTests. It's in-code self-documentation. Leaving a little nugget of your understanding of the functionality behind for other developers in the future. Even if a method is horribly named or the business logic is so complex it takes an hour to understand exactly what is happening, a simple group of (input -> method -> expected output) unit test can get you working with the method fairly fast.
Actually, if I am extending a giant god function that is un-tested. I find it exceedingly helpful to add some tests for the specific use cases I am adding, just to be sure I have understood the code path relevant to my change correctly.
The weird part is not that you don't write unit tests, plenty of developers don't.
It's that in 25+ years, you never even had the stray thought "hm, maybe all this manual work that I keep repeating might be worth automating?" and then tested it out to see whether it was worth doing or not
The manual testing really isn't very hard... where it relies on the same logic I use existing logic. Where it's new logic, particularly new UI, testing really can't be automated. Where it's not UI but server logic or DB queries, it's pretty explicit and tested six ways to Sunday before committing. I think the only true unit tests I ever wrote, if you can call them that, were to test pseudorandom behavior across distributions of things like character spawning in games, where player testing wouldn't be sufficient to spot anomalies. For something like that where you're going to have a billion variants, it makes sense to test for oddities on some subset of a million or two. In other words, tests to make sure that output fit a certain curve. Not that logic itself worked or didn't work, because if I can think of a test to write where it might fail, I'd already have fixed it not to fail under that test.
There is a difference between these three statements:
1. It can't be tested.
2. I don't know how to test it.
3. The tests don't give enough value given the effort required to write them.
The less experience you have with tests, the worse the cost-benefit analysis comes out. This is why people disagree over unit tests. Good tests give a huge productivity gain, but the opposite is true of bad tests. Unfortunately, there's a vicious cycle:
You write bad tests
-> You don't value tests
-> You don't spend time learning to improve tests
-> You continue to write bad tests
It's also good for incrementally building stuff. e.g. I have 3 scenarios I want to get working, I write a test for the first and get it passing. Then I move onto scenario 2, this requires changing the code slightly for scenario 1, but I don't have to re-test scenario 1 as I've got a unit test that is running each time I change the code that tells me it still passes, and so scenario 1 is still working. Then when I do scenario 3, I know 1 and 2 are still working without manually going back and testing them...
> Then you make a change, that change impacts 10 of those. You now need to go back and re-test, manually, those 10 thing and check they haven't broke.
Unit tests provide no guarantee that those 10 things haven't broke. They can break in a way that's not under test. Or they can be working individually, but breaking as a whole.
You still have to test those 10 things manually before merging.
Sure, sometimes unit tests will show you regression early on. Early feedback can be useful. But it will never give you confidence, it doesn't eliminate manual testing.
The important part is that you're making a change and you don't have confidence that it is safe. This is a signal of bad design. In most cases the right solution is to fix the design, not throw tests at the problem. There are exceptions of course, but even in those cases tests are not something to celebrate or be proud of.
It's like getting obese and then celebrating statin therapy. I mean it's great, but not as great as not being obese in the first place.
We used to use a similar approach for testing page objects in stb-tester. It was called stbt auto-selftest (it has since been removed). It would run your page objects against canned screenshots and generate a Python doctest capturing the behaviour of this page objects. You’d then commit the doctest to git and have ci set up to check correctness. See https://david.rothlis.net/pageobjects/characterisation-tests
This was really helpful - we could then see in or pull requests how changes to our page object code would affect the behaviour of that code - just look at the diffs.
It worked ok for a while, but we soon hit limitations with this approach. The big one is the interaction with git. We would get many merge conflicts in these files, particularly when rebasing to reorder commits. Sometimes we’d forget to regenerate the files and have to rerun and rebase after submitting to ci. It worked okish for us as we are sophisticated git users comfortable with rebase and filter-branch, and we’d designed the system so could debug issues effectively, but it wasn’t suitable for wider use.
We’ve moved away from this model to a slightly different one. We still generate this data[^1], but we store the data outside of git in a database indexed by git commit sha. We then display this data and associated diffs in a web-ui. We use GitHub checks to wire it back up to pull requests, so it’s just as visible as when it was committed to git, but more convenient and displayed in a nicer format. You can use the same web-ui to approve changes to this data.
[^1]: though in more structured form than doctests
There were other advantages to our new approach too: because all the running of the code is happening server side it is both easier and faster.
Easier because you don’t need to have a complete development environment available on the machine that you’re making the change. You can make a quick change in the GitHub web ui on mobile and see the results. This is particularly important in our case as we have some code that depends on specific hardware to run (CUDA), that isn’t just available on any developer machine.
Faster because the server can run the process in parallel across a farm of machines, which will be faster than running locally.
On a related note: I think that deterministic generating something based on a git commit and then later diffing the results across commits is in general a very useful technique. Our builds work similarly. We build our software into (several) container/disk images, then we show the diff between these build artefacts on PRs.
It can help you spot when something changes that you didn’t expect, and vice-versa.
For this to work we store all our builds indexed by git commit sha.
This does not look joyful at all. The second code block example is almost impossible for me to read and understand. A typical unittest with assertions at the end is much easier.
"In most testing frameworks I’ve used, even the simplest assertions require a surprising amount of toil. Suppose you’re writing a test for a fibonacci function. You start writing assert fibonacci(15) == ... and already you’re forced to think. What does fibonacci(15) equal?"
You need to know what you expect the program to do before writing a test verifying that it does it, no? I don't understand what the author is trying to say here.
Maybe that coding is sometimes used to figure out stuff you yourself do not know ffs. I understand your point, but what we do as SWEs is the secondary function of coding.
You'd write a test like that in RSpec if you want to reuse it for other examples. The other example they present in Ruby, is what you would write if you were using Ruby std lib testing library.
It doesn't make the test any better or more "joyful".
Also the REPL-like experience their library has actually sucks.
I don't see any value, and if you don't find writing tests to be a joyful experience before that library was created, I don't see how you will see once you have that library.
People normally write tests for just the sake of correctness. I treat unit test as real debugging environment, it's debugging experience that makes writing test code enjoyful as i see.
I’m currently losing the will to live with integration tests for an http service I’m writing.
It calls out to other services (often 3 or 4 external api calls for each of my http resources, each with complex data structures).
(I just want the external API data in a database!)
Tests are a pain because of the matrix of fixtures needed. My API endpoints behave very differently based on the attributes of the objects in the response from the other APIs. So I have lots of somewhat complex code to generate fixture data, which again is another source of error and a whole lot of code.
For one of my endpoints, I have to mock 1-6 API calls, multiplied by 3 or 4 variations of the returned data. It’s torture.
None of my tests are that obvious what it’s testing because the fixture code and json assertions are so big and long.
We aren’t using an http mocking lib right now (eg wiremock) but I’m wondering if it’s worth it.
I guess a lot of it is because the APIs I’m calling are so badly designed.
I think it probably IS the APIs that are a problem but since you usually cannot instantly get rid of such things I think you need tricks.
e.g. keeping your fixtures in separate files and naming them in such a way that the actual data doesn't appear in your test code.
You might use some fixture as a base value (in my case I had this problem with a big old structure that describes PKI certificates of may types) and have derived fixtures that are (the base plus a change to the "account" field or the base with the "accept" field set to "false"). I did this with python classes and inheritance. Each derived class copied the base class's data and applied a diff. YMMV but I mention it just in case it helps.
If these are all named and kept in separate files then your actual integration tests can look a bit simpler perhaps?
> I guess a lot of it is because the APIs I’m calling are so badly designed.
Sort of, sounds like the problem is that you have a distributed monolith. If things are this tightly coupled they should just live in same process. If you even had just a regular monolith (which I don't recommend but it is much better than distributed monolith) you wouldn't have this particular problem. Of course another alternative is to redesign the system so that services are based on independent domain capabilities so that they have high cohesion and loose coupling.
Seeing as how you prob can’t consolidate apis to a single process for more control.
Seriously, having an http mocking tool is so helpful. I use msw, a node equivalent of wiremock. To help with a situation like yours, it’s easy to say “before this executes, I want the mock tool to return this response”. The abstractions are good enough that I don’t have to deal with recreating my dep-injected service objects, and really reduces the tests’ lines of code. It also seems like the most complex bit here is handling your permutations of code paths, as well as their error paths. I salute. I can’t think of a way to simplify it all except to be glad I work on smaller products now that don’t require so many microservices or 3rd party apis.
The unfortunate bit is ensuring your mock stays up to date with their api. But if we have most or all of the variations, then it’s their fault for breaking their APIs on us, no?
It's a bit different in that it'll save the expected to a different file... IMHO that's usually nicer because the test result is usually big and having it separated makes more sense.
When rerunning it's possible to run pytest with '--force-regen' and then check the git diff to see if all the changes were expected.
I find go test writing a misery because of the style golang recommends and our company has decoded is to be enforced. You have a list of data values and expected results in an array and then a test at the bottom that applies these array items to the function you're testing - sounds great right? No - because often the only way to get all the variations you need in the test is to have helper functions that are referenced in the array to setup the situations you want. i.e. it ends up being far harder to add a new and slightly different test to this array than to just write a new test - and that's not supposed to be allowed.
Since you brought up Go, I had a question as someone who's just started using the language. Do Go users generally just use "testing" package for writing tests or are there other commonly used packages that can be used in place of or in tandem with that? Just trying to get a sense of what the options are and what the "state of the art" is.
The standard testing package for the win, and many pull in a light weight assertion library like Testify.
I love testing in Go (and _loath_ testing in Ruby). Our unit and integration tests are mostly great. Tests are just code. No DSL or special anything; just normal code. My favorite test that showed me the light: it would spin up several smtp servers (the system under test), like a dozen or more, get the SMTP conversation to different points and wait for timeouts and ensure that everything behaved. The test ran in under 10ms.
We have tests that ensure logs happened, metrics emitted, timeouts honored, graceful shutdowns work, and error paths are easily validated with test fakes that return canned errors (no mocks). I love testing in Go.
Nice, thanks for the reply. Our Python tests run incredibly slowly. Having fast tests makes such a big difference. It can be really demoralizing waiting for a bunch of tests to finish. In general, I'm incredibly excited for how fast Go is, especially compared to Python. I had almost forgotten what fast feels like.
With python you could choose to use Pypy and make it work for your system - that might speed things up.
Alternatively, you could use smoketests. In my last largish project we grouped the tests into a short run and long run group. This allowed us to develop code and do a set of short tests in a fairly quick cycle and then before merging to the master branch we would do more comprehensive tests or let the CI system do it.
FWIW I got more joy from python by a long way but go isn't that awful except when you're trying to fork a github module to make it do something different.
Oh ya know, I never did consider PyPy. Is it significantly faster than standard CPython? I'm going to look into trying it out. Would be amazing if we could easily get performance improvements with it.
Also, your idea about having a smaller set of quick running tests and then separate smoke tests is actually a great idea. What we've really done in many of the cases where we should have a unit tests, is essentially to write integration tests. Not quite end to end but pretty close and that's where a lot of the slowness comes from.
Most of the non-joyful experience of writing tests lies in the context preparation, not so much in the expected outputs.
I think we need a modern version of design by contract instead. Its one of the most powerful techniques of testing software is unfortunately often ignored and overlooked these days...
Some less known things about design by contract:
- you can easily test way more than types in preconditions and postconditions
- extreme web controller postcondition example: every time there is a 200 response to a POST /new request, there should be at least one document more in storage than there was before the request. (caveat: unless there are deletes - then sum the deletes up too)
- things important to the business look a ton like contracts
- e.g. a diff of two legal documents always contains the full content (list of words) of both documents (never accidentally swallow content!)
- your postconditions can surprisingly often fully describe the desired function behavior
- with sufficient amount of pre and postconditions, your property tests look like "exercise the module or entire system with random actions" -> unit and integration tests
It can also be unit tests. You can take any function that has contracts defined and run a fuzzer (or a property testing generator) on its inputs, and its an isolated unit test.
Or you could generate input actions for the entire system to exercise all contracts, and get e2e tests.
Or you could run your code in production and exercise contracts (maybe at a certain sampling rate) to get observability and "testing in production".
Its the most powerful and flexible concept I know of, but it requires thinking in properties (pre/postconditions) which can be a bit tricky.
I agree, but unit tests are cheaper, you can do them unilaterally, and you can accomplish a lot with them in an organization that can't or won't invest in integration testing.
Even when integration tests exist, they typically don't go beyond exercising each ability of the system one or two ways. They don't achieve good "data coverage" or branch coverage, which is what property-based unit testing excels at.
I’m meaning to write a library that’s the intersection of design by contract and abstract algebra for Python. You’d be able to say that getting a diff is associative and get free tests just for documenting that.
This is not always in your control - especially if you're using libraries and frameworks. But also there are sometimes complex functions that (need to) combine lots complex data (say from multiple APIs) together to produce results, and there is not much that can be done about it.
Take a structured document diff for example - its seemingly simple, diff(documentRepresentation1, documentRepresentation2) -> diffDocument. But in actual reality it has to handle all kinds of edge cases in the structure of the inner documents, and preparing the structure of those documents is hard enough that you actually need to build helper functions to make it easier to make a variety of them.
Careful, you're coming dangerously close to reinventing SystemVerilog's formal verification support :) SV formal is tough at first but wonderful once you really get it because it lets you lay out your constraints and preconditions and then the solver verifies that your circuit fulfills those requirements.
I don’t have sufficient experience to judge whether you are correct or not. But I hope that you are correct!
I think code deserves way more assertions and validations. And (like you alluded to) fine-grained ways to turn them on or off; we shouldn’t shy away from expensive tests that might take hundreds of milliseconds just because they might be non-practical to run everywhere—instead we should have configuration to turn them on or off. And not just simple on/off assertions like in Java but things that can have metadata like “cost”, “priority”, and so on.
And, of course, some things (probably the actual contracts) might be always-on.
Writing tests is a normal experience for me (just like writing code) as long as the tests make sense. As soon as some manager (or senior engineer) start to ask for X% code coverage, then it all goes to hell and writing tests become a robotic task that adds nothing but a burden to maintain in the future.
ps: this wasn't a joke, a lot of daily activities are 'chores' because of the context. You can almost always make it fun, more useful, more fulfilling. But society doesn't invest in that. Which leads to bore/burnout
127 comments
[ 2.6 ms ] story [ 187 ms ] threadI.e. For all x, Fib(x+2)=Fib(x)+Fib(x+1)
A property based test produces values of x and tests this property (or invariant).
If it fails the framework usually then tries to find the simplest failing case.
That said, there are plenty of cases where your reaction is justified, where the test is implemented the same way as the function it's testing when the correctness property could be specified more simply.
Property based tests fuzz test values you may not have dreamed of. Given a string as the required type it could generate some exotic stuff.
Even outside that special case it is always possible to generate just some input data and check for the absence of runtime errors or exceptions. That approach can be improved by adding assertions, pre- and postconditions to your code under test.
Additionally, there a different ways to come up with good properties, see: https://www.youtube.com/watch?v=zvRAyq5wj38
* Test the externally visible useful properties of your system, rather than implementation details. Related: don’t bother with mocks unless there’s no other way
* Keep your tests DRY: if there’s some invariant you want to check have it appear once in your test code
* Don’t over-test: you can delete tests and checks that fully overlap with other tests
* If you need to over-test make it easy to regenerate the expected results from your implementation (characterisation tests) as espoused by the article. * the amount of testing effort should be proportional to how important the functionality is. Reduce the effort for low impact features
I'm still on the fence about that but I'm not sure how else to test something that is essentially a glorified algebra formula.
- Server-side rendering and testability: https://david.rothlis.net/server-side-rendering-good/
- Testing "Page Objects" (classes written for UI tests): https://david.rothlis.net/pageobjects/characterisation-tests
Humans are better at verifying the answer is correct over coming up with the answer.
The thing that makes test writing unjoyous for me is most code is hard to test, usually requiring all kinds of scaffolding and even resources like files, db, or mocked http requests.
Being Jane Street they probably use OCaml so yes functional programming languages are nicer to write tests in.
I feel like the Jane Street approach of embedding the snapshots directly in test sources would make a qualitative difference to the ergonomics. The post implies that this requires some emacs integration, but there's no reason it couldn't be done by a test runner process instead. It would also be cool to have an LSP that provide a hover text or go-to-definition for externally stored snapshots.
> usually requiring all kinds of scaffolding and even resources like files, db, or mocked http requests
Yes, anything with timestamps or generated identifiers gets very annoying as you have to manually exclude certain data from the snapshots, negating some of the "easy wins" a snapshot/approval process provides for pure/idempotent code.
Unless you're doing TDD or chatGPT is too busy, there's almost no excuse now to not write some unit tests ;)
Usually I get good tests from ChatGPT when I approach it as an iterative process, requesting multiple improvements to the generated test based on what it gives to me. Note that it doesn't replace the skillsets you need to know to write good test coverage.
For example, you can ask to generate integration tests instead of unit tests in case it need context. Providing details on how the testing code should be generated really helps. Also, asking to refactoring the code in preparation to make it testeable, for example, or requesting to convert some functions to actual pure functions, or requesting to refactor a piece of code it generated to a separate function. Then you ask to generate tests for normal and also for boundary conditions. The more specific you get, the chance of getting a good and extensive tests from it is much higher.
Having the tests and code generated by ChatGPT really helps to catch the subtle bugs it usually generates on the generated code (fixing it manually), usually I get test coverage that proof the robustness I needed for production code.
This approach still needs manual fine tuning of the generated code, I think ChatGPT still struggle to get the context right but in general, when it makes sense to use it, I'm more productive writing tests in this way than manually.
In my day job, I write Ruby, and it didn't impress me much when I used it with Rspec. I'd says it was saving me maybe a dozen or so keystrokes in total to write a new spec.
My job is to first consider a dozen ways a new feature might be used; then consider how it could be misused; then write a backend portion and UI for it, testing it for usability and potential for error along the way, using everything I've already thought could go wrong; then monitor it once it's deployed to see if anything actually does go wrong. After actually testing it through everything I can think of by hand, function by function, line by line as I write it, it seems totally stupid to write code to test my own code.
Not to sound like a shit, but that's what end users are for ;)
About how long would you say that process takes you? Do you repeat it every time a minor change needs to happen to your code? If not, how do you verify the impact of your changes?
Usually though, in practice, this looks like tracing/logging several times in the middle of each function to make sure I'm getting it right. Where unit tests are supposedly most handy is when unexpected types are passed to functions, leading to unexpected errors. The majority of proofing against that can be done by strict typing and good code documentation.
Ultimately the best test is when you ask yourself "wait, what if someone sent this" and you try it to see if you can break your own code. That's just a spidey sense in the back of your head. If you didn't have the specific doubt to begin with, I don't know how you could write a unit test to disprove it anyway.
[edit] just a note to any new coders reading this: Spend the time to always read, read, read everything about how to harden the code you write for security purposes. Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.
I am sorry if I am being a bit mean - I just have had to take over maintenance of things like this and have found it a nightmare from hell so far.
My main concern, of course, is not leaving a mess for myself. That should be every coder's priority. Then taking over someone else's projects wouldn't be so hard.
Nevertheless, I highly recommend having automated tests and running them in the presence of memory profilers etc like valgrind if you're using a compiled memory unsafe language like C or C++.
I would look for another job if I had to work with someone that refused to write them to be honest. It's not that I love tests, but I have had to fix other people's bugs.
I can't speak for others but that's precisely how and why I write tests. It goes like "Wait, what if someone sent this" -> check what happens -> write a test to automate that check -> optionally, fix the code to handle this specific case.
Then, over time you accumulate tests that check all these weird edge cases for you and it doesn't take much to run them over and over again every time you change the code.
> Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.
I agree with the first part but not the conclusion. My point is that with both manual and automated testing, you still needs to think. It's just that automated tests let you build an executable knowledge base of all the edge cases, errors, security issues etc. you've thought about in the past, and run them with every code change. Hence the value.
Not everyone is as diligent and not everyone will have thought through/remembered all those edge cases when they happen to end up maintaining/adding to your code.
Personally I find that tests are a great way to ensure all that hard work you did thinking about those edge cases isn’t wasted. That and manually testing stuff is annoying after the first time.
Tests are useful when you change code that is unrelated, but still somehow connected. You would never notice the issue as you don't manually try the unrelated code, but the test would catch it.
If you had unit/integration tests for all potential edgecases you came up with, you don't have to manually test that over and over.
Well, I get it. You're right in principle. The truth is, though, I have had to rewrite software from scratch in new languages every 10 years, and these sorts of bugs only appear a few times per decade. Once a piece of central business logic works, it usually works forever; a change to that central logic will of course require changes and tests across the whole system, from user inputs to annual reports, but that can't be helped. I suspect you'd then have to rewrite the unit tests, too.
It's probably true that when I die, most of my software will slowly wind down and eventually be abandoned. But also that's probably true even if I were to write a lot of tests.
This is a textbook example of something that property based testing can catch. Yes, the tests are smarter than the person that wrote the code. At least, smarter in a different way or maybe it’s that the people who wrote the property based testing framework are super smart.
Look up QuickCheck for the OG or property based testing + your language.
I don’t exactly understand your bug, but the way it would work is something like you’d make some generators that create night stays and rates and put those together into a bill. Then you test the property that the number of line items on the bill is the number of nights + number of discounts (or whatever is right). It will search to break this property and if it breaks it, find a minimal case. Seriously check it out, it’s perfect for this sort of thing.
Edge cases could be due to accidental complexity. In this case, the solution is to remove accidental complexity, not to freeze it forever in tests.
Edge cases could be due to essential complexity. In this case, if people are making changes without remembering them, it means they are changing code they don't understand. Sure, tests are making it easier. I don't want it to be easier.
> tests are a great way to ensure all that hard work you did thinking about those edge cases isn’t wasted
It is also a great way to ensure that your best talent is wasted on building guard rails.
Think about it: your best engineers are spending time making worst engineers more productive. And they are not even doing it through mentoring, so your worst engineers will remain where they are.
I think a lot of people miss the (in my opinion) best use case for UnitTests. It's in-code self-documentation. Leaving a little nugget of your understanding of the functionality behind for other developers in the future. Even if a method is horribly named or the business logic is so complex it takes an hour to understand exactly what is happening, a simple group of (input -> method -> expected output) unit test can get you working with the method fairly fast.
Actually, if I am extending a giant god function that is un-tested. I find it exceedingly helpful to add some tests for the specific use cases I am adding, just to be sure I have understood the code path relevant to my change correctly.
The weird part is not that you don't write unit tests, plenty of developers don't. It's that in 25+ years, you never even had the stray thought "hm, maybe all this manual work that I keep repeating might be worth automating?" and then tested it out to see whether it was worth doing or not
1. It can't be tested.
2. I don't know how to test it.
3. The tests don't give enough value given the effort required to write them.
The less experience you have with tests, the worse the cost-benefit analysis comes out. This is why people disagree over unit tests. Good tests give a huge productivity gain, but the opposite is true of bad tests. Unfortunately, there's a vicious cycle:
And what happens in most companies? We hired a junior engineer and don't have work for them... let them write tests!
Then you make a change, that change impacts 10 of those. You now need to go back and re-test, manually, those 10 thing and check they haven't broke.
That's mainly why I use unit tests, as regression testing.
Also as documentation, if I'm unsure around what a piece of code is actually doing, it's helpful to see tests that hit and find something like.
When_FooIsInThisState_ThenIExpectThisThingToHappen()
Then see the setup and expected output.
It's also good for incrementally building stuff. e.g. I have 3 scenarios I want to get working, I write a test for the first and get it passing. Then I move onto scenario 2, this requires changing the code slightly for scenario 1, but I don't have to re-test scenario 1 as I've got a unit test that is running each time I change the code that tells me it still passes, and so scenario 1 is still working. Then when I do scenario 3, I know 1 and 2 are still working without manually going back and testing them...
They can really save a lot of manual effort.
Unit tests provide no guarantee that those 10 things haven't broke. They can break in a way that's not under test. Or they can be working individually, but breaking as a whole.
You still have to test those 10 things manually before merging.
Sure, sometimes unit tests will show you regression early on. Early feedback can be useful. But it will never give you confidence, it doesn't eliminate manual testing.
The important part is that you're making a change and you don't have confidence that it is safe. This is a signal of bad design. In most cases the right solution is to fix the design, not throw tests at the problem. There are exceptions of course, but even in those cases tests are not something to celebrate or be proud of.
It's like getting obese and then celebrating statin therapy. I mean it's great, but not as great as not being obese in the first place.
This was really helpful - we could then see in or pull requests how changes to our page object code would affect the behaviour of that code - just look at the diffs.
It worked ok for a while, but we soon hit limitations with this approach. The big one is the interaction with git. We would get many merge conflicts in these files, particularly when rebasing to reorder commits. Sometimes we’d forget to regenerate the files and have to rerun and rebase after submitting to ci. It worked okish for us as we are sophisticated git users comfortable with rebase and filter-branch, and we’d designed the system so could debug issues effectively, but it wasn’t suitable for wider use.
We’ve moved away from this model to a slightly different one. We still generate this data[^1], but we store the data outside of git in a database indexed by git commit sha. We then display this data and associated diffs in a web-ui. We use GitHub checks to wire it back up to pull requests, so it’s just as visible as when it was committed to git, but more convenient and displayed in a nicer format. You can use the same web-ui to approve changes to this data.
[^1]: though in more structured form than doctests
Easier because you don’t need to have a complete development environment available on the machine that you’re making the change. You can make a quick change in the GitHub web ui on mobile and see the results. This is particularly important in our case as we have some code that depends on specific hardware to run (CUDA), that isn’t just available on any developer machine.
Faster because the server can run the process in parallel across a farm of machines, which will be faster than running locally.
It can help you spot when something changes that you didn’t expect, and vice-versa.
For this to work we store all our builds indexed by git commit sha.
We don't write code to tell computers what to do.
We write code to communicate with other humans. And we spend more time reading (other people's) code than writing it.
You need to know what you expect the program to do before writing a test verifying that it does it, no? I don't understand what the author is trying to say here.
You'd write a test like that in RSpec if you want to reuse it for other examples. The other example they present in Ruby, is what you would write if you were using Ruby std lib testing library.
It doesn't make the test any better or more "joyful".
Also the REPL-like experience their library has actually sucks.
I don't see any value, and if you don't find writing tests to be a joyful experience before that library was created, I don't see how you will see once you have that library.
[1] https://github.com/Teemu/pytest-sugar
I’m currently losing the will to live with integration tests for an http service I’m writing.
It calls out to other services (often 3 or 4 external api calls for each of my http resources, each with complex data structures).
(I just want the external API data in a database!)
Tests are a pain because of the matrix of fixtures needed. My API endpoints behave very differently based on the attributes of the objects in the response from the other APIs. So I have lots of somewhat complex code to generate fixture data, which again is another source of error and a whole lot of code.
For one of my endpoints, I have to mock 1-6 API calls, multiplied by 3 or 4 variations of the returned data. It’s torture.
None of my tests are that obvious what it’s testing because the fixture code and json assertions are so big and long. We aren’t using an http mocking lib right now (eg wiremock) but I’m wondering if it’s worth it.
I guess a lot of it is because the APIs I’m calling are so badly designed.
e.g. keeping your fixtures in separate files and naming them in such a way that the actual data doesn't appear in your test code.
You might use some fixture as a base value (in my case I had this problem with a big old structure that describes PKI certificates of may types) and have derived fixtures that are (the base plus a change to the "account" field or the base with the "accept" field set to "false"). I did this with python classes and inheritance. Each derived class copied the base class's data and applied a diff. YMMV but I mention it just in case it helps.
If these are all named and kept in separate files then your actual integration tests can look a bit simpler perhaps?
Sort of, sounds like the problem is that you have a distributed monolith. If things are this tightly coupled they should just live in same process. If you even had just a regular monolith (which I don't recommend but it is much better than distributed monolith) you wouldn't have this particular problem. Of course another alternative is to redesign the system so that services are based on independent domain capabilities so that they have high cohesion and loose coupling.
Seriously, having an http mocking tool is so helpful. I use msw, a node equivalent of wiremock. To help with a situation like yours, it’s easy to say “before this executes, I want the mock tool to return this response”. The abstractions are good enough that I don’t have to deal with recreating my dep-injected service objects, and really reduces the tests’ lines of code. It also seems like the most complex bit here is handling your permutations of code paths, as well as their error paths. I salute. I can’t think of a way to simplify it all except to be glad I work on smaller products now that don’t require so many microservices or 3rd party apis.
The unfortunate bit is ensuring your mock stays up to date with their api. But if we have most or all of the variations, then it’s their fault for breaking their APIs on us, no?
https://pypi.org/project/pytest-regressions/
It's a bit different in that it'll save the expected to a different file... IMHO that's usually nicer because the test result is usually big and having it separated makes more sense.
When rerunning it's possible to run pytest with '--force-regen' and then check the git diff to see if all the changes were expected.
I love testing in Go (and _loath_ testing in Ruby). Our unit and integration tests are mostly great. Tests are just code. No DSL or special anything; just normal code. My favorite test that showed me the light: it would spin up several smtp servers (the system under test), like a dozen or more, get the SMTP conversation to different points and wait for timeouts and ensure that everything behaved. The test ran in under 10ms.
We have tests that ensure logs happened, metrics emitted, timeouts honored, graceful shutdowns work, and error paths are easily validated with test fakes that return canned errors (no mocks). I love testing in Go.
Alternatively, you could use smoketests. In my last largish project we grouped the tests into a short run and long run group. This allowed us to develop code and do a set of short tests in a fairly quick cycle and then before merging to the master branch we would do more comprehensive tests or let the CI system do it.
FWIW I got more joy from python by a long way but go isn't that awful except when you're trying to fork a github module to make it do something different.
Also, your idea about having a smaller set of quick running tests and then separate smoke tests is actually a great idea. What we've really done in many of the cases where we should have a unit tests, is essentially to write integration tests. Not quite end to end but pretty close and that's where a lot of the slowness comes from.
Appreciate the reply!
I think we need a modern version of design by contract instead. Its one of the most powerful techniques of testing software is unfortunately often ignored and overlooked these days...
Some less known things about design by contract:
- you can easily test way more than types in preconditions and postconditions
- things important to the business look a ton like contracts - your postconditions can surprisingly often fully describe the desired function behavior- with sufficient amount of pre and postconditions, your property tests look like "exercise the module or entire system with random actions" -> unit and integration tests
(see Hilel Wayne's excellent video on this topic https://www.youtube.com/watch?v=MYucYon2-lk)
Some things that could really help modernize design by contract
- Test at a certain sampling rate in production for expensive contracts (do not turn them off)
- Report contract violations via observability mechanisms (tracing, sentry etc)
- Better language support for contract DSLs
Or you could generate input actions for the entire system to exercise all contracts, and get e2e tests.
Or you could run your code in production and exercise contracts (maybe at a certain sampling rate) to get observability and "testing in production".
Its the most powerful and flexible concept I know of, but it requires thinking in properties (pre/postconditions) which can be a bit tricky.
Even when integration tests exist, they typically don't go beyond exercising each ability of the system one or two ways. They don't achieve good "data coverage" or branch coverage, which is what property-based unit testing excels at.
The right fix here is to simplify the context preparation - make the functions more self contained and less dependent on external scope.
Take a structured document diff for example - its seemingly simple, diff(documentRepresentation1, documentRepresentation2) -> diffDocument. But in actual reality it has to handle all kinds of edge cases in the structure of the inner documents, and preparing the structure of those documents is hard enough that you actually need to build helper functions to make it easier to make a variety of them.
My overall happiness with testing improved significantly once I started using it.
https://github.com/assert-value/assert_value_elixir
I think code deserves way more assertions and validations. And (like you alluded to) fine-grained ways to turn them on or off; we shouldn’t shy away from expensive tests that might take hundreds of milliseconds just because they might be non-practical to run everywhere—instead we should have configuration to turn them on or off. And not just simple on/off assertions like in Java but things that can have metadata like “cost”, “priority”, and so on.
And, of course, some things (probably the actual contracts) might be always-on.
There’s a lot of exciting potential!
WHACK!