I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand.
So rather than try to learn to black boxes at the same time , I fall back to "several more unit tests to document more edge cases to defensibly guard against"
Is there some simple way to describe this defensive programming iteration pattern in Hypothesis? Normally we just null-check and return early and have to deal with the early-return case. How do I quickly write property tests to check that my code handles the most obvious edge cases?
For working with legacy systems I tend to start with Approval Tests (https://approvaltests.com/). Because they don't require me to understand the SUT very well before I can get started with them, and because they help me start to understand it more quickly.
When I'm "up to my armpits in trying to understand the on-the-ground problem", I find PBT great for quickly find mistakes in the assumptions/intuitions I'm making about surrounding code and helper functions.
Whenever I find myself thinking "WTF? Surely ABC does XYZ?", and the code for ABC isn't immediately obvious, then I'll bang-out an `ABC_does_XYZ` property and see if I'm wrong. This can be much faster than trying to think up "good" examples to check, especially when I'm not familiar with the domain model, and the relevant values would be giant nested things. I'll let the computer have a go first.
It’s been quite some time since I’ve been in the business of writing lots of unit tests, but back in the day, I found hypothesis to be a big force multiplier and it uncovered many subtle/embarrassing bugs for me. Recommend. Also easy and intuitive to use.
I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (it was always the 25th item that failed) quickly led me to find the bug. Once my property tests were running overnight and not finding any failures after thousands and thousands of random cases, I started to feel a lot more confident that I'd nailed down the bugs.
Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent.
(I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a library of useful strategies without people calling foul in code review is obnoxious.)
> Once my property tests were running overnight and not finding any failures
Quickly addressing the time it takes to run property-based tests, especially in Python: it's extremely helpful to make sure that running subsets of unit tests is convenient and documented and all your developers are doing it. Otherwise they may revolt against property-based testing because of the time it takes to run tests. Again, especially in Python.
I can no longer edit the parent comment, but I want to address one thing:
> Once my property tests were running overnight
This was happening because I was setting my test runner to say "Keep generating random data and running tests for 8 hours, no matter how many tests that is", not because the tests were slow. I was running literally millions of tests overnight (usually about 2-3 million), though later when I added some slow tests that number went down into the tens of thousands of tests overnight.
I may have given the impression that property-based testing is slow. It doesn't have to be. It can be very, very quick. But the more random tests you run, the more likely to are to uncover the really, REALLY bizarre edge cases.
It seems to only implement a half of QuickCheck idea, because there is no counterexample shrinking. Good effort though! I wonder how hard would it be to derive generators for any custom types in python - probably not too hard, because types are just values
1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does. Any logical error that might be slipped into your SUT implementation has a high risk of also slipping into your test and will therefore be hidden by the complexity, even though it would be obvious from just looking at a few well thought through examples.
2. Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of. You basically just give up and leave edge case finding to chance. Testing for 0 or -1 or 1-more-than-list-length are obvious cases which both you the human test writer and some test framework can easily generate, and they are often actual edge cases. But what really constitutes an edge case depends on your implementation. You as the developer know the implementation and have a chance of coming up with the edge cases. You know the dark corners of your code. Random tests are just playing the lottery, replacing thinking hard.
Your comment is downvoted currently, but I think it has value in the discussion (despite being wrong in literally every respect) because it shows the immense misleading power of a single extraordinarily poorly chosen headline example on the project page. Testing a sorting function against the results of the sorted builtin is concise, and technically correct, but (even though there are situations where it would be exactly the right thing to do) it is completely misleading to anyone new to the concept when it comes to indicating what property-based testing is all about.
> It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it.
It does not, and it would be next to worthless if it did. It requires being able to define the properties required of the SUT and right implement code that can refute them if they are not present (the name "hypothesis" for this library is, in fact, a reference to that; PBT treats the properties of code as a hypothesis, and attempts to refute it.)
> but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does.
No, see my reimplementation of the sorting example without resorting to the builtin (or any) sorting function other than the one being tested:
> Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of.
You may think this based on zero experience, but I have seen no one who has tried hypothesis even once who has had that experience. Its actually very good at finding edge cases.
> You as the developer know the implementation and have a chance of coming up with the edge cases.
You as the developer have a very good chance of finding the same edge cases when writing tests for your own code that you considered when writing the code. You have much less chance of finding edge case when writing tests that you missed when writing code. You can incorporate the knowledge of probable edge cases you have when crafting Hypothesis tests just as with more traditional unit tests—but with traditional unit tests you have zero chance of finding the edge cases you didn't think of. Hypothesis is, actually, quite good at that.
> Random tests are just playing the lottery, replacing thinking hard.
Property-based testing doesn’t replace the thinking that goes into traditional unit testing, it just acts as a force multiplier for it.
(Hypothesis maintainer here) If you have recommendations for a better example on the front page, I'd love to hear them! (I mean this entirely genuinely and non-sarcastically; I agree sorting can give misleading ideas, but it is also concise and well understood by every reader).
I've taught this in my testing courses. I find that (pytest) fixtures are often good enough for coming up with multiple tests but are simple enough to implement.
My theory is that only code written in functional languages has complex properties you can actually test.
In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.
I think core of the problem in property-based testing that the property/specification needs to be quite simple compared to the implementation.
I did some property-based testing in Haskell and in some cases the implementation was the specification verbatum. So what properties should I test?
It was clearer where my function should be symmetric in the arguments or that there is a neutral element, etc..
If the property is basically your specification which (as the language is very expressive) is your implementation then you're just going in circles.
I strongly disagree, but I think there are a few problems
1. Lots of devs just don't know about it.
2. It's easy to do it wrong and try and reimplement the thing you're testing.
3. It's easy to try and redo all your testing like that and fail and then give up.
Here's some I've used:
Tabbing changes focus for UIs with more than 1 element. Shift tabbing the same number of times takes you back to where you came from.
This one on TVs with u/d/l/r nav -> if pressing a direction changes focus, pressing the opposite direction takes you back to where you came from.
An extension of the last -> regardless of the set of API calls used to make the user interface, the same is true.
When finding text ABC in a larger string and getting back `Match(x, start, end)`, if I take the string and chop out string[start:end] then I get back exactly ABC. This failed because of a dotted I that when lowercased during a normalisation step resulted in two characters - so all the positions were shifted. Hypothesis found this and was able to give me a test like "find x in 'İx' -> fail".
No input to the API should result in a 500 error. N, where N>0, PUT requests result in one item created.
Clicking around the application should not result in a 404 page or error page.
Overall I think there's lots of wider things you can check, because we should have UIs and tools that give simple rules and guarantees to users.
I think testing culture in general is suffering because the most popular styles/runtimes don’t support it easily.
Most apps (at least in my part of the world) these days are absolutely peppered with side effects. At work our code is mostly just endpoints that trigger tons of side effects, then return some glob of data returned from some of those effects. The joys of micro services!!
If you’re designing from the ground up with testing in mind, you can make things super testable. Defer the actual execution of side effects. Group them together and move local biz logic to a pure function. But when you have a service that’s just a 10,000 line tangle of reading and writing to queues, databases and other services, it’s really hard to ANY kind of testing.
I think that’s why unit testing and full on browser based E2E testing are popular these days. Unit testing pretends the complexity isn’t there via mocks, and lets you get high test coverage to pass your 85% coverage requirement. Then the E2E tests actually test user stories.
I’m really hoping there’s a shift. There are SO many interesting and comprehensive testing strategies available that can give you such high confidence in your program. But it mostly feels like an afterthought. My job has 90% coverage requirements, but not a single person writes useful tests. We have like 10,000 unit tests literally just mocking functions and then spying on the mocked return.
For anybody wanting to see a super duper interesting use of property based testing, check out “Breaking the Bank with test contract”, a talk by Allen Rohner. He pretty much uses property based testing to verify that mocks of services behave identically to the actual services (for the purpose of the program) so that you can develop and test against those mocks. I’ve started implementing a shitty version of this at work, and it’s wicked cool!!
Probably because it's difficult to build simplistic intuition around them in languages that don't have robust built-in mechanisms that enable that kind of thinking.
For example, gen-testing in Clojure feels simple and intuitive, but if I have to do something similar, hmmm... I dunno, say in Java, I wouldn't even know where to start - the code inevitably would be too verbose; mutable state would obscure invariants; and the required boilerplate alone would cloud the core idea. While in Clojure, immutability-by-default feels transformative for gen-testing; data generation is first-class - generators are just values or functions; composition is natural; debugging is easier; invariants are verifiable; FP mindset already baked in - easier to reason about stateful sequences/event streams; there's far less ceremony overall;
But if I'm forced to write Java and see a good case for prop-based testing, I'd still probably try doing it. But only because I already acquired some intuition for it. That's why it is immensely rewarding to spend a year or more learning an FP language, even if you don't see it used at work or every day.
So, the honest answer to your question would probably be: "because FP languages are not more popular"
The Python survey data (https://lp.jetbrains.com/python-developers-survey-2024/) holds pretty consistently at 4% of Python users saying they use it, which isn't as large as I'd like, but given that only 64% of people in the survey say they use testing at all isn't doing too badly, and I think certainly falsifies the claim that Python programs don't have properties you can test.
I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why.
Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all.
Obviously it's just a tiny example, but in general I often find writing a test that will cover every hypothetical case isn't realistic.
Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it?
I've used hypothesis in practice for testing a slugify function. When someone signs up for our product our software produces a subdomain based on the organisation name you input. People put all sorts of weird things in that field, and we want to be pretty broad in what we'll accept.
The hypothesis test for this is near trivial. Pass some text to the slugify function, and check it either a) throws an invalid input error (e.g. input too short etc) or b) return a string that's a valid domain name.
Doing this found me so many odd little edge cases around length truncation and idna encoding. The one bug I would've never found myself is the fact that lowercasing "ß" turns it into "ss" which broke truncation, resulting in an invalid domain only if it includes ß and is exactly 64 characters long.
hypothesis is a pretty niche tool, but boy when it's the right tool it's the right tool.
I'm using sqlglot to parse hundreds of old mysql back up files to find diffs of schemas. The joys of legacy code. I've found hypothesis to be super helpful for tightening up my parser. I've identified properties (invariants) and built some strategies. I can now generate many more permutations of DDL than I'd thought of before. And I have absolutely confidence in what I'm running.
I started off TDD covered the basics. Learned what I needed to learn about the files I'm dealing with, edge cases, sqlglot and then I moved onto Hypothesis for extra confidence.
I'm curious to see if it'll help with commands for APIs. I nothing else it'll help me appreciate how liberal my API is when perhaps I don't want it to be?
One cool application of this is schemathesis. I really enjoyed it, and I found more input validation bugs in my code than I can count.
Very useful for public APIs.
Great project. I used it a lot, but now I mostly prefer ad hoc generators. Hypothesis combinators quickly become unmaintainable mess for non-trivial objects. Also, shrinking is not such a big deal when you can generate your data in a complexity-sorted order.
My first impression on property-based testing is that it's likely to introduce flakes that distract engineers from their task at hand, forcing them to investigate a new failed test in an area that they weren't even touching.
Never heard of “property-based testing” before. Coming from Go, I mostly use table and fuzzy tests.
Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language.
I am a huge fan of the "Explanations" page of their docs: "These explanation pages are oriented towards deepening your understanding of Hypothesis, including its design philosophy."
I feel much more software documentation could greatly benefit from this approach. Describing the "why" and design tradeoffs helps me grok a system far better than the typical quickstart or tutorials which show snippets but offer little understanding. That is, they rarely help me answer: is this going to solve my problem and how?
37 comments
[ 4.2 ms ] story [ 60.5 ms ] threadSo rather than try to learn to black boxes at the same time , I fall back to "several more unit tests to document more edge cases to defensibly guard against"
Is there some simple way to describe this defensive programming iteration pattern in Hypothesis? Normally we just null-check and return early and have to deal with the early-return case. How do I quickly write property tests to check that my code handles the most obvious edge cases?
So if you already have parametrized tests, you're already halfway there.
Whenever I find myself thinking "WTF? Surely ABC does XYZ?", and the code for ABC isn't immediately obvious, then I'll bang-out an `ABC_does_XYZ` property and see if I'm wrong. This can be much faster than trying to think up "good" examples to check, especially when I'm not familiar with the domain model, and the relevant values would be giant nested things. I'll let the computer have a go first.
Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent.
(I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a library of useful strategies without people calling foul in code review is obnoxious.)
Quickly addressing the time it takes to run property-based tests, especially in Python: it's extremely helpful to make sure that running subsets of unit tests is convenient and documented and all your developers are doing it. Otherwise they may revolt against property-based testing because of the time it takes to run tests. Again, especially in Python.
> Once my property tests were running overnight
This was happening because I was setting my test runner to say "Keep generating random data and running tests for 8 hours, no matter how many tests that is", not because the tests were slow. I was running literally millions of tests overnight (usually about 2-3 million), though later when I added some slow tests that number went down into the tens of thousands of tests overnight.
I may have given the impression that property-based testing is slow. It doesn't have to be. It can be very, very quick. But the more random tests you run, the more likely to are to uncover the really, REALLY bizarre edge cases.
1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does. Any logical error that might be slipped into your SUT implementation has a high risk of also slipping into your test and will therefore be hidden by the complexity, even though it would be obvious from just looking at a few well thought through examples.
2. Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of. You basically just give up and leave edge case finding to chance. Testing for 0 or -1 or 1-more-than-list-length are obvious cases which both you the human test writer and some test framework can easily generate, and they are often actual edge cases. But what really constitutes an edge case depends on your implementation. You as the developer know the implementation and have a chance of coming up with the edge cases. You know the dark corners of your code. Random tests are just playing the lottery, replacing thinking hard.
> It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it.
It does not, and it would be next to worthless if it did. It requires being able to define the properties required of the SUT and right implement code that can refute them if they are not present (the name "hypothesis" for this library is, in fact, a reference to that; PBT treats the properties of code as a hypothesis, and attempts to refute it.)
> but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does.
No, see my reimplementation of the sorting example without resorting to the builtin (or any) sorting function other than the one being tested:
https://news.ycombinator.com/item?id=45825482
> Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of.
You may think this based on zero experience, but I have seen no one who has tried hypothesis even once who has had that experience. Its actually very good at finding edge cases.
> You as the developer know the implementation and have a chance of coming up with the edge cases.
You as the developer have a very good chance of finding the same edge cases when writing tests for your own code that you considered when writing the code. You have much less chance of finding edge case when writing tests that you missed when writing code. You can incorporate the knowledge of probable edge cases you have when crafting Hypothesis tests just as with more traditional unit tests—but with traditional unit tests you have zero chance of finding the edge cases you didn't think of. Hypothesis is, actually, quite good at that.
> Random tests are just playing the lottery, replacing thinking hard.
Property-based testing doesn’t replace the thinking that goes into traditional unit testing, it just acts as a force multiplier for it.
Why is it not more popular?
My theory is that only code written in functional languages has complex properties you can actually test.
In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.
Property, fuzzy, snapshot testing. Great tools that make software more correct and reliable.
The challenge for most developers is that they need to change how they design code and think about testing.
I’ve always said the hardest part of programming isn’t learning, it’s unlearning what you already know…
I did some property-based testing in Haskell and in some cases the implementation was the specification verbatum. So what properties should I test? It was clearer where my function should be symmetric in the arguments or that there is a neutral element, etc..
If the property is basically your specification which (as the language is very expressive) is your implementation then you're just going in circles.
1. Lots of devs just don't know about it.
2. It's easy to do it wrong and try and reimplement the thing you're testing.
3. It's easy to try and redo all your testing like that and fail and then give up.
Here's some I've used:
Tabbing changes focus for UIs with more than 1 element. Shift tabbing the same number of times takes you back to where you came from.
This one on TVs with u/d/l/r nav -> if pressing a direction changes focus, pressing the opposite direction takes you back to where you came from.
An extension of the last -> regardless of the set of API calls used to make the user interface, the same is true.
When finding text ABC in a larger string and getting back `Match(x, start, end)`, if I take the string and chop out string[start:end] then I get back exactly ABC. This failed because of a dotted I that when lowercased during a normalisation step resulted in two characters - so all the positions were shifted. Hypothesis found this and was able to give me a test like "find x in 'İx' -> fail".
No input to the API should result in a 500 error. N, where N>0, PUT requests result in one item created.
Clicking around the application should not result in a 404 page or error page.
Overall I think there's lots of wider things you can check, because we should have UIs and tools that give simple rules and guarantees to users.
Most apps (at least in my part of the world) these days are absolutely peppered with side effects. At work our code is mostly just endpoints that trigger tons of side effects, then return some glob of data returned from some of those effects. The joys of micro services!!
If you’re designing from the ground up with testing in mind, you can make things super testable. Defer the actual execution of side effects. Group them together and move local biz logic to a pure function. But when you have a service that’s just a 10,000 line tangle of reading and writing to queues, databases and other services, it’s really hard to ANY kind of testing.
I think that’s why unit testing and full on browser based E2E testing are popular these days. Unit testing pretends the complexity isn’t there via mocks, and lets you get high test coverage to pass your 85% coverage requirement. Then the E2E tests actually test user stories.
I’m really hoping there’s a shift. There are SO many interesting and comprehensive testing strategies available that can give you such high confidence in your program. But it mostly feels like an afterthought. My job has 90% coverage requirements, but not a single person writes useful tests. We have like 10,000 unit tests literally just mocking functions and then spying on the mocked return.
For anybody wanting to see a super duper interesting use of property based testing, check out “Breaking the Bank with test contract”, a talk by Allen Rohner. He pretty much uses property based testing to verify that mocks of services behave identically to the actual services (for the purpose of the program) so that you can develop and test against those mocks. I’ve started implementing a shitty version of this at work, and it’s wicked cool!!
Probably because it's difficult to build simplistic intuition around them in languages that don't have robust built-in mechanisms that enable that kind of thinking.
For example, gen-testing in Clojure feels simple and intuitive, but if I have to do something similar, hmmm... I dunno, say in Java, I wouldn't even know where to start - the code inevitably would be too verbose; mutable state would obscure invariants; and the required boilerplate alone would cloud the core idea. While in Clojure, immutability-by-default feels transformative for gen-testing; data generation is first-class - generators are just values or functions; composition is natural; debugging is easier; invariants are verifiable; FP mindset already baked in - easier to reason about stateful sequences/event streams; there's far less ceremony overall;
But if I'm forced to write Java and see a good case for prop-based testing, I'd still probably try doing it. But only because I already acquired some intuition for it. That's why it is immensely rewarding to spend a year or more learning an FP language, even if you don't see it used at work or every day.
So, the honest answer to your question would probably be: "because FP languages are not more popular"
The Python survey data (https://lp.jetbrains.com/python-developers-survey-2024/) holds pretty consistently at 4% of Python users saying they use it, which isn't as large as I'd like, but given that only 64% of people in the survey say they use testing at all isn't doing too badly, and I think certainly falsifies the claim that Python programs don't have properties you can test.
I haven't used it, or property-based testing, but I can see how it could be useful.
[1]: https://github.com/flyingmutant/rapid
Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all.
Obviously it's just a tiny example, but in general I often find writing a test that will cover every hypothetical case isn't realistic.
Anyone here use this testing in the wild? Where's it most useful? Do you have the issue I described? Is there an easy way to overcome it?
The hypothesis test for this is near trivial. Pass some text to the slugify function, and check it either a) throws an invalid input error (e.g. input too short etc) or b) return a string that's a valid domain name.
Doing this found me so many odd little edge cases around length truncation and idna encoding. The one bug I would've never found myself is the fact that lowercasing "ß" turns it into "ss" which broke truncation, resulting in an invalid domain only if it includes ß and is exactly 64 characters long.
hypothesis is a pretty niche tool, but boy when it's the right tool it's the right tool.
I started off TDD covered the basics. Learned what I needed to learn about the files I'm dealing with, edge cases, sqlglot and then I moved onto Hypothesis for extra confidence.
I'm curious to see if it'll help with commands for APIs. I nothing else it'll help me appreciate how liberal my API is when perhaps I don't want it to be?
Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language.
[1]: https://github.com/flyingmutant/rapid
[2]: Oh, I just found out about testing/quick.
I feel much more software documentation could greatly benefit from this approach. Describing the "why" and design tradeoffs helps me grok a system far better than the typical quickstart or tutorials which show snippets but offer little understanding. That is, they rarely help me answer: is this going to solve my problem and how?