I see the point of pytest and the greatness of it. I like pytest but I like minimalism more. I dont find it a good showcase here: couldn't you have kept only the asserts and just run the script?
In that case, only the first fail will be reported and exit. Which is fair enough in some cases, but may be confusing for long sequences of tests.
(There’s the option to get that behaviour in pytest as well with a parameter. Also, pytest may run the tests in parallel, which may speed up the results)
Test driven development is just such a good habit to get into. You trade just a little bit of ramp-up speed at the beginning of an implementation for a massive reduction in cognitive overhead.
I'd love to see how test driven development looks like for ML systems.
You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?
It's doable once you're out of pure experimentation and into the development phase at which test driven development can help.
Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys.
Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an object of type X (a trained sklearn object say, or dictionary of weights that can be deserialised), fails loudly if you don't have enough samples from category Y etc.
Test that the predict method returns a probability as a float, a predicted class as an int, a DataFrame with metadata and headers, etc etc.
assert test_performance > threshold ? This would only be a sanity check for big deviations though. For more subtle changes one may need to do a proper statistical test. I believe the jury is still out of how to do that properly for deep ML models - challenge is the lack of independence in CV folds and generally the compute time it takes to evaluate.
However, I would probably not do performance check inside a unit-testing framework. Instead treat this as quality indicators like performance benchmarks, code coverage etc. It may be a "gate", that needs to pass to allow a new model into production.
To evaluate performance over time, one would preferably want labeled datasets for test gathered at different points in time. Which requires a (reliable) continuous labeling process.
One can also gather customer feedback about performance, track those as metrics.
These things are probably more in the "monitoring" part of a system, rather than unit-testing time though.
You can test on a tester of use some of the examples from the checklist paper. In that paper they might add "I hate you" to some random data and assert that sentiment doesn't improve.
Those are statistics/ML questions, not software development ones, so software development processes like TDD are only tangentially relevant really.
But in any case, it's actually fairly easy to test that your trained model has sufficient accuracy: choose a metric, choose a threshold for said metric, and check that the observed metric on a testing set (data that the model was not trained on) is above the desired threshold. Repeat for several different metrics for a better understanding of how well the model performs. This can be put in a set of unit tests.
Check residuals, inspect the logical implications of the regression coefficients (or whatever), plot a few curves etc to be more sure again. This can't really be put in unit tests, nor should it be. But again, this is more statistics than software development.
Same goes for model deterioration - every so often you check that the metric(s) still beat the minimum threshold on more recent data.
The stated problem requires returning the smallest integer greater than 0 in the list. 0 will only be the running minimum if only non-positive numbers have been seen so far, and since i is greater than zero in the condition, if minimum == 0 it means i is the first positive integer we’ve seen, so even though i > minimum (i > 0) we need to swap i as the minimum at that point.
How would you write tests for the following problem with random output?
Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers.
Example implementation
def biasedcoin(n,p):
import randrange from random
heads = tails = 0
for i in range(n):
if randrange(100) < 100*p:
heads += 1
else:
tails += 1
return heads/tails
Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result.
Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the two-digit bias number.
Otherwise you are just testing randrange itself.
If you are asking how to test a pseudorandom number generator, you have a few choices. You can either fix the random seed and test the algorithm’s precise implementation on a large number of known results. Or you can define tests statistically with margins of error, for example testing the entropy in a series of uniformly generated bits or the resulting distribution in a series of random draws from a fixed list, and decide what level of precision is tolerable before considering a test failed.
Thanks for your answer. With mocking, what happens if someone writes their if condition as: (if randrange(1000) < 1000*p) or worse, (if randrange(1000) > (1000-1000p).
I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions.
I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.
Sure, this is simple! You provide them with a definition written in Coq and make them implement a proof that their implementation satisfies that definition. Use probabilistic couplings to make sure the output of the biased coin function adheres to specified distribution.
You can easily implement a grader that checks the the proof.
> Simple: use mocker.patch to patch out random.randrange and have it return a fixed sequence of results that would imply a known result.
Thank you for this. Another way to put this is "your tests for code that uses a 3rd party API should not amount to an uptime test for the 3rd party API."
Some time ago, I observed that tests in the Boost.Accumulators library were similarly confounding tests for the code one wrote vs tests of statistical hypotheses:
Mocks should be used when the call you depend on is hard to get to behave in the ways you need for your test.
"Randrange" isn't hard to get to behave in the way you want. In fact, the way you want it to behave here is all it does!
I'd argue if what you want is to do engineering, mocking this isn't what you'd want to do. You want to ensure your code works. And your test can tell you that it works, even if "randrange" works differently while maintaining the same signature after you update the module it came from.
If you're doing software engineering for aviation for example, I doubt "well, my tests using mocks passed, it's just that the dependency broke it's contract" is a good enough excuse for a catastrophe.
No, mocks should be used for any kind of external dependency, whether it is the external OS system calls, built in library functions that interface with sockets, databases, etc. It’s not about whether the resource you are patching is easy / hard to deal with, for example the pytest built in fixture tmpdir abstracts this patching for temporary directories and filesystem ops even though that stuff is “easy” and behaves. Anytime you rely on something outside the runtime environment (even if through a stdlib) you should patch it.
You should have a completely different set of tests (integration tests / end to end tests) that exercises important unmocked validation points. And your test runner should allow you to seamlessly switch between the two sets or combine them.
For example, use @pytest.mark.integration to distinguish all tests needed unmocked dependencies, and have some “integration-tests.ini” config for that.
Then “pytest” runs all the tests with mocks (runs fast, tests logical correctness with tight feedback) and “pytest -c integration-tests.ini” runs all tests or runs the subset requiring real third party resource access. It can run slower, sometimes fail for flaky reasons like network blip, etc.
Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better.
This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward.
At best I could say in some unique use cases there could be API trade offs that make this ok, but definitely not at all for passing around pseudorandom sampler functions. Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
> Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better.
It gives users the option to do that, but doesn't burden them. This can be an implementation detail, whilst the public API provides a default like `bias = lambda: findBias(random.randrange)`. Since it's Python we could also use a default argument, e.g.
import random
def findBias(randrange=random.randrange):
heads = tails = 0
for i in range(n):
if randrange(100) < 100*p:
heads += 1
else:
tails += 1
return heads/tails
> This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward.
This is overly-simplistic and disingenuous:
If other functions don't care about this parameter, then they don't need to use it, since it's just an implementation detail; they'll call the public API, which will use the default random.randrange, and these callers don't even have to know that the parameter exists. This scenario is strictly an improvement to what you describe; heck, we could even mock 'randrange' like you describe and it would work in exactly the same way! (except we don't need to, since we can just do normal dependency injection via function arguments instead)
If other functions do care about this parameter, e.g. because they want to pass along a parameter from their own callers, then that's solving a different problem. In this case it's even better to be passing around parameters: mocking is a last-resort crutch when we're unable to refactor legacy code, but even in those cases it should only be used during testing. It's a very bad idea to be monkey-patching the standard library during the normal course of a program run; especially when it's just to avoid adding an extra parameter to a function we control.
> Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
I know a lot of this is subjective, but I'd like to point out that the mocking solution essentially works using mutable global variables. If we override the name "random.randrange", we don't actually know what else we might be affecting; for all we know, our bias-calculator could invoke a bunch of helper libraries which happen to rely on certain behaviour from random.randrange, which mocking will break in unexpected ways. In contrast, changing which value we pass as a function argument will not break arbitrary other code far-away which just-so-happens to be using the previous value. That's the fundamental problem with mocking, and why it should be avoided except as a last resort: it forces us to make big, inappropriate assumptions about not only the code we're testing, but also everything else it might ever interact with. The way pytest implements that mocking (by mutating globals like 'random.randint') just makes this even worse.
First, if you can ensure the seed of the generator, then that could be used on a case by case basis. But this tricky, since it introduces a subtle dependency on the implementation of biasedcoin.
Another option is to split the function/process into a stochastic and deterministic part. Test the deterministic part thoroughly. biasedcoin is too trivial for this, but it works nicely in more complicated setups.
For biasedcoin, I would probably go with a statistical approach. Given a certain bias, the expected value and distribution is known. I would simply test that several trials of the function lie within some bound.
The statistical approach definitely gives a developer more confidence in the correctness of a solution and guards against regressions. Though any test that checks the outcome of a random process against statistical measures is expected to fail occasionally.
For example, if the test flips the coin 1,000 times, there is better than a 99% chance that the outcome would result in 450 to 550 heads. So if you write the test using `450 <= heads <= 550`, you know that it will fail ~1% of the time. And if you expand the range to reduce the rate of false negatives, you have reduced the confidence the test is validating correctness.
Having said that, I still find statistical tests to be very helpful when building out code that uses randomness. However, these tests typically do not make it into the CI/CD pipeline.
In practice there is no such tradeoff, the tails of a Gaussian fall off extremely rapidly so if you simply 10x the number of iterations you’ll have failure become astronomically unlikely.
If it were a non-trivial example I would split it in two: One function to generate the data, and one function to count and do the statistics. Then you can pass the second one known datasets and verify it gives you correct statistics, and at least be confident that that part is correct.
Of course that still leaves the generation and randomness part. You could make a separate function for numberToTailOrHead(...) that you can test. Or maybe you should have the ability to send in which source of randomness it uses. And then you can provide a fake random implementation that tests the edge cases (0, 0.5, 0.999).
Side remark, but I really wish interviewers gave me the option of a terminal with vim since that’s pretty much standard on any Unix machine I might ssh into these days.
All too often someone puts me in some weird IDE and I feel like a cat with boots on.
Someone on our school's group chat shared they had a technical interview where she was asked to write code on a shared google doc. I really hope that was an outlier.
I told them flat out no, gave them my ip, made an account on my local machine and started a shared tmux session over ssh. Since that's my usual ide I didn't lose any capability.
Depends on the ISP, my mobile dongle has a dynamic public ip, I have a little twitter bot that sends out a tweet every time its ip changes.
On my wired connection I have 1tb symmetric fiber with 16 static ipv4 ips and an arbitrarily large number of ipv6 addresses, I just need to register more if I need them (which I really don't).
When testing solutions to algorithmic problems, it's often useful to use randomized property checking to verify that the solution's expected properties hold for approximately all inputs. The QuickCheck family of testing tools is probably the best known application of this approach. It's also pretty easy to roll your own. Some hand-rolled examples in Python:
> unittest is an old horse and cart, while pytest is the batmobile.
I really wish more was said than just this. Why is it that pytest is often preferred and unittest is considered a bad choice? What is the impact on the quality of the tests?
44 comments
[ 3.3 ms ] story [ 92.4 ms ] thread(There’s the option to get that behaviour in pytest as well with a parameter. Also, pytest may run the tests in parallel, which may speed up the results)
You write the test for your non-existent model, then write the model and try to train the model to 'pass' the test? Do you also train it on the test case or on other data only?
Test that this ETL function expects a DataFrame with a given schema and returns one with a different (but also known) schema, even with all these edge cases in the filters and group-bys.
Test that the "train_classifier" method/function rejects negative penalisation parameters, returns an object of type X (a trained sklearn object say, or dictionary of weights that can be deserialised), fails loudly if you don't have enough samples from category Y etc.
Test that the predict method returns a probability as a float, a predicted class as an int, a DataFrame with metadata and headers, etc etc.
However, I would probably not do performance check inside a unit-testing framework. Instead treat this as quality indicators like performance benchmarks, code coverage etc. It may be a "gate", that needs to pass to allow a new model into production.
To evaluate performance over time, one would preferably want labeled datasets for test gathered at different points in time. Which requires a (reliable) continuous labeling process. One can also gather customer feedback about performance, track those as metrics. These things are probably more in the "monitoring" part of a system, rather than unit-testing time though.
But in any case, it's actually fairly easy to test that your trained model has sufficient accuracy: choose a metric, choose a threshold for said metric, and check that the observed metric on a testing set (data that the model was not trained on) is above the desired threshold. Repeat for several different metrics for a better understanding of how well the model performs. This can be put in a set of unit tests.
Check residuals, inspect the logical implications of the regression coefficients (or whatever), plot a few curves etc to be more sure again. This can't really be put in unit tests, nor should it be. But again, this is more statistics than software development.
Same goes for model deterioration - every so often you check that the metric(s) still beat the minimum threshold on more recent data.
(Also, it’s part of the standard library.)
https://docs.python.org/3/library/doctest.html
Write a function biasedcoin(n,p) that takes the number of coin flips, n, and the probability of heads, p. It flips a biased coin n times, and returns the ratio of number of heads/number of tails. p is guaranteed to have two significant numbers eg. p = 0.60 or p = 0.74. You should use the random.randrange function to generate random numbers.
Example implementation
def biasedcoin(n,p):
Your test should not depend on the internal workings of randrange or anything relying on unfixed random state. Your test is only checking if, given correct results from randrange (or any other external world source of random draws) that the rest of your function correctly produces the two-digit bias number.
Otherwise you are just testing randrange itself.
If you are asking how to test a pseudorandom number generator, you have a few choices. You can either fix the random seed and test the algorithm’s precise implementation on a large number of known results. Or you can define tests statistically with margins of error, for example testing the entropy in a series of uniformly generated bits or the resulting distribution in a series of random draws from a fixed list, and decide what level of precision is tolerable before considering a test failed.
I am trying to write tests to autograde some HW assignments, and so I really have to think adversarially and imagine my students writing the weirdest yet correctly working functions.
I really haven't come up with a better answer than checking the statistics of the returned value by running the function a whole bunch of times, like you suggest.
You can easily implement a grader that checks the the proof.
Thank you for this. Another way to put this is "your tests for code that uses a 3rd party API should not amount to an uptime test for the 3rd party API."
Some time ago, I observed that tests in the Boost.Accumulators library were similarly confounding tests for the code one wrote vs tests of statistical hypotheses:
https://www.nu42.com/2016/12/cpp-boost-median-test.html
"Randrange" isn't hard to get to behave in the way you want. In fact, the way you want it to behave here is all it does!
I'd argue if what you want is to do engineering, mocking this isn't what you'd want to do. You want to ensure your code works. And your test can tell you that it works, even if "randrange" works differently while maintaining the same signature after you update the module it came from.
If you're doing software engineering for aviation for example, I doubt "well, my tests using mocks passed, it's just that the dependency broke it's contract" is a good enough excuse for a catastrophe.
You should have a completely different set of tests (integration tests / end to end tests) that exercises important unmocked validation points. And your test runner should allow you to seamlessly switch between the two sets or combine them.
For example, use @pytest.mark.integration to distinguish all tests needed unmocked dependencies, and have some “integration-tests.ini” config for that.
Then “pytest” runs all the tests with mocks (runs fast, tests logical correctness with tight feedback) and “pytest -c integration-tests.ini” runs all tests or runs the subset requiring real third party resource access. It can run slower, sometimes fail for flaky reasons like network blip, etc.
Why not just pass in 'randrange' as a function argument?
It's amazing to me how many complicated and convoluted hacks people can come up with, to avoid calling a function with an argument.This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward.
At best I could say in some unique use cases there could be API trade offs that make this ok, but definitely not at all for passing around pseudorandom sampler functions. Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
It gives users the option to do that, but doesn't burden them. This can be an implementation detail, whilst the public API provides a default like `bias = lambda: findBias(random.randrange)`. Since it's Python we could also use a default argument, e.g.
> This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward.This is overly-simplistic and disingenuous:
If other functions don't care about this parameter, then they don't need to use it, since it's just an implementation detail; they'll call the public API, which will use the default random.randrange, and these callers don't even have to know that the parameter exists. This scenario is strictly an improvement to what you describe; heck, we could even mock 'randrange' like you describe and it would work in exactly the same way! (except we don't need to, since we can just do normal dependency injection via function arguments instead)
If other functions do care about this parameter, e.g. because they want to pass along a parameter from their own callers, then that's solving a different problem. In this case it's even better to be passing around parameters: mocking is a last-resort crutch when we're unable to refactor legacy code, but even in those cases it should only be used during testing. It's a very bad idea to be monkey-patching the standard library during the normal course of a program run; especially when it's just to avoid adding an extra parameter to a function we control.
> Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.
I know a lot of this is subjective, but I'd like to point out that the mocking solution essentially works using mutable global variables. If we override the name "random.randrange", we don't actually know what else we might be affecting; for all we know, our bias-calculator could invoke a bunch of helper libraries which happen to rely on certain behaviour from random.randrange, which mocking will break in unexpected ways. In contrast, changing which value we pass as a function argument will not break arbitrary other code far-away which just-so-happens to be using the previous value. That's the fundamental problem with mocking, and why it should be avoided except as a last resort: it forces us to make big, inappropriate assumptions about not only the code we're testing, but also everything else it might ever interact with. The way pytest implements that mocking (by mutating globals like 'random.randint') just makes this even worse.
First, if you can ensure the seed of the generator, then that could be used on a case by case basis. But this tricky, since it introduces a subtle dependency on the implementation of biasedcoin.
Another option is to split the function/process into a stochastic and deterministic part. Test the deterministic part thoroughly. biasedcoin is too trivial for this, but it works nicely in more complicated setups.
For biasedcoin, I would probably go with a statistical approach. Given a certain bias, the expected value and distribution is known. I would simply test that several trials of the function lie within some bound.
For example, if the test flips the coin 1,000 times, there is better than a 99% chance that the outcome would result in 450 to 550 heads. So if you write the test using `450 <= heads <= 550`, you know that it will fail ~1% of the time. And if you expand the range to reduce the rate of false negatives, you have reduced the confidence the test is validating correctness.
Having said that, I still find statistical tests to be very helpful when building out code that uses randomness. However, these tests typically do not make it into the CI/CD pipeline.
Of course that still leaves the generation and randomness part. You could make a separate function for numberToTailOrHead(...) that you can test. Or maybe you should have the ability to send in which source of randomness it uses. And then you can provide a fake random implementation that tests the edge cases (0, 0.5, 0.999).
All too often someone puts me in some weird IDE and I feel like a cat with boots on.
Are other companies copying them to the Tee?
I told them flat out no, gave them my ip, made an account on my local machine and started a shared tmux session over ssh. Since that's my usual ide I didn't lose any capability.
But I have to have my ip publicly accessable (permission from isp?) beforehand right?
On my wired connection I have 1tb symmetric fiber with 16 static ipv4 ips and an arbitrarily large number of ipv6 addresses, I just need to register more if I need them (which I really don't).
That said I didn't get the job, so caveat emptor.
https://github.com/tmoertel/practice/blob/master/dailycoding...
https://github.com/tmoertel/practice/blob/master/dailycoding...
https://github.com/tmoertel/practice/blob/master/dailycoding...
I really wish more was said than just this. Why is it that pytest is often preferred and unittest is considered a bad choice? What is the impact on the quality of the tests?
- lets you use regular assert statements, rather than assertEqual, assertTrue, etc.
- does not impose the use of classes
- the way it does fixtures just feels nicer to use than unittest
Basically, it's more ergonomic and IME more flexible.