This is neat, it reminds me of Hypothesis which does randomised property-based testing and if it finds a failing edge case it saves it to a database so that it's always used in future test runs.
That said, I suspect I'd find a similar limitation with this as with Hypothesis. Most of the tests I write aren't for pure functions, or require a fair amount of setup or complex assertions. It's possible to write test helpers that do all of this, and I do, but too much of that means tests that are complex enough to need their own tests, so I think it's important to balance.
This is probably specific to my normal work (Python, web services), but I'd suspect applies to a lot of testing done elsewhere.
> Most of the tests I write aren't for pure functions
In response to this, I recommend the "Functional Core, Imperative Shell"[0] talk/pattern. The idea is to extract your business logic as pure functions independent of the data access layer. This pattern allowed me to test large portions of a code base using property tests. This works really well in most cases and gives me much more confidence in the product that will be deployed.
While the pattern is good to always be striving for, I think it can be too much boilerplate at times.
If I've got say 100 lines of business logic, it makes sense to have 10 lines to do the necessary data shuffling to get everything in order so that you can keep your business logic a nice pure function.
On the other hand, if you have 5 lines of business logic, it maybe doesn't make sense to have 10 lines of shuffling data to be able to keep the business logic a pure function. Not so much for the time to write it but because it can obscure the functionality for the reader.
This feels like it defeats the purpose of writing tests though. For me writing test very much has to do with validating my assumptions... it helps me gain confidence in my code. Now I'd have to trust these automated tests? My confidence just wouldn't be there.
Makes sense, but it's always good to capture the intent of the test. If it is to prevent a regression, having data about the issue that was fixed is vital.
In general, I prefer to start writing tests, just enough the tests fail but call the functionality (this makes API design flaws obvious), then actually build the functionality and, finally, make the test conditions reflect what we need.
Title says unit tests. But to GP's point, it isn't really automatic, the developer still decides what to test. It is transforming REPL logs, not generating tests based directly on the code (which isn't useful in my experience).
To be honest, my development REPL logs have a lot of junk, so it would be extra work to write a clean one, and then make the right assertion (maybe I was looking for "even number" not "exactly 132"). Lisp-y REPLs are friendlier than some, but it would be cool to associate it with something like a notebook where it is easier to update imports/mocks/etc out of order. Cool idea though.
Your problem appears to be your entire focus is pointlessly nitpicking the meaning of everyone’s words where enough context exists to trivially disambiguate it.
This is more, did I accidentally break something? The correctness of the tests is not known but you will know which tests have changed. Then you can go and review that code and see why it changed and whether the new behavior is correct. This gives you confidence in a change didn’t break anything but not that things weren’t broken from the start.
To me, testing in the REPL is also about validating my assumptions.
REPL-driven development is basically test-driven development with sorter feedback cycles during active development, but the downside is you need to make an extra effort to go back and preserve the important tests for posterity. This might well reduce some of that friction.
Oftentimes, finagling with the REPL is merely about getting the syntax right ("did I use the right number of parens?") rather than formulating assumptions. In such cases, using this tool doesn't invalidate the TDD workflow, since your assumptions don't change.
The essential complexity of writing unit tests is in formulating your assumptions ahead of time. As you correctly point out, no tool can ever solve this. However, an accidental complexity is correctly expressing your assumptions in code. Tools such as a REPL and this library can definitely help with that.
It is tough. On the one hand, I agree with you. On the other , writing tests seems both incredibly inefficient as well as missing the intended purpose. You cannot have a test suite which ‘proves’ the implementation is correct. A test suite is really like Swiss cheese - there are so many holes that implementation code can get through.
Now, bear in mind that I say this as a passionate tester and verification enthusiast. I don’t have a better alternative, so I keep writing tests. And yes, they do catch bugs. I’m not saying they have no value. I’m saying, they always let some bugs through, I.e, they are not complete.
And that makes the cost of them very depressing. Not only cost to write, cost to continuously run.
This brings to mind a conversation that I've had several times over the years. It goes something like this:
> Wouldn't it be cool if we could have code that would test itself?
> Yeah, if only there were some kind of automated way to encapsulate the requirements of classes and methods that could ensure that they behaved within some known boundaries...?
We've had automated unit tests for quite some time, but they are more often called compiler checks. Want your "automated unit tests" to be more comprehensive? Improve your type system and your compiler!
With all that said, I do not want to be dismissive of peoples' projects. This is fun and neat.
Type checking, compile-time errors won't check if the program you wrote is correct, only that it's internally consistent. That is an entirely different thing than checking for correctness.
You do need a layer of tests that actually try to prove that given some input your program actually does what it's supposed to do. That is hard to automate.
That's untrue. There are type-systems and type-checkers which can check for correctness, as specified by a mathematical formula. This is proof assistants such as Coq[1] work. For example, take a look at fiat-crypto [2], a project that generates code that is correct by mathematical proof, and is now deployed in Chrome. Another famous example is the CompCert [3] compiler, a C compiler along with a correctness proof that the compiler generates assembly which correctly simulates the C semantics.
These are based on powerful types as "dependent types" which can encode mathematics and programs in the same programming language, and allow the mathematics to "talk about" the program. The mathematical underpinning is the Curry-Howard-Lambek correspondence.
I have yet to see an effective version of these "wonder" tools actually work in practice. That said, they make great snake oil for testing automation teams that often end up doing manual testing because the promises of the "easy" automation were never fulfilled.
Pretty good. Essentially just a way to save function executions and replay them later against previous evaluation results.
I would guess that Jupyter notebooks could easily be made into such a thing as well. Also doctests with editor integration might work like this already?
32 comments
[ 2.9 ms ] story [ 88.6 ms ] threadThat said, I suspect I'd find a similar limitation with this as with Hypothesis. Most of the tests I write aren't for pure functions, or require a fair amount of setup or complex assertions. It's possible to write test helpers that do all of this, and I do, but too much of that means tests that are complex enough to need their own tests, so I think it's important to balance.
This is probably specific to my normal work (Python, web services), but I'd suspect applies to a lot of testing done elsewhere.
In response to this, I recommend the "Functional Core, Imperative Shell"[0] talk/pattern. The idea is to extract your business logic as pure functions independent of the data access layer. This pattern allowed me to test large portions of a code base using property tests. This works really well in most cases and gives me much more confidence in the product that will be deployed.
[0]: https://www.destroyallsoftware.com/screencasts/catalog/funct...
While the pattern is good to always be striving for, I think it can be too much boilerplate at times.
If I've got say 100 lines of business logic, it makes sense to have 10 lines to do the necessary data shuffling to get everything in order so that you can keep your business logic a nice pure function.
On the other hand, if you have 5 lines of business logic, it maybe doesn't make sense to have 10 lines of shuffling data to be able to keep the business logic a pure function. Not so much for the time to write it but because it can obscure the functionality for the reader.
In general, I prefer to start writing tests, just enough the tests fail but call the functionality (this makes API design flaws obvious), then actually build the functionality and, finally, make the test conditions reflect what we need.
To be honest, my development REPL logs have a lot of junk, so it would be extra work to write a clean one, and then make the right assertion (maybe I was looking for "even number" not "exactly 132"). Lisp-y REPLs are friendlier than some, but it would be cool to associate it with something like a notebook where it is easier to update imports/mocks/etc out of order. Cool idea though.
1. Unit testing is a kind of functional testing. On a unit.
2. And regression testing is rerunning your existing functional tests after, say, refactoring, to see if they pass.
Complex? Confusing? Which part of this precisely is confusing?
So if I say 'I wrote a unit test' what kind of test do you expect?
The headline says they're generating unit tests automatically.
Which means I expect that I run a command, and it plops out unit test code. automatically. without my interaction.
The article is about recording repl interactions. that's not automatic.
and really, they look like implementation details? so bad examples? I don't know scheme, so shrug
You gotta pick a better hobby.
REPL-driven development is basically test-driven development with sorter feedback cycles during active development, but the downside is you need to make an extra effort to go back and preserve the important tests for posterity. This might well reduce some of that friction.
The essential complexity of writing unit tests is in formulating your assumptions ahead of time. As you correctly point out, no tool can ever solve this. However, an accidental complexity is correctly expressing your assumptions in code. Tools such as a REPL and this library can definitely help with that.
Now, bear in mind that I say this as a passionate tester and verification enthusiast. I don’t have a better alternative, so I keep writing tests. And yes, they do catch bugs. I’m not saying they have no value. I’m saying, they always let some bugs through, I.e, they are not complete.
And that makes the cost of them very depressing. Not only cost to write, cost to continuously run.
> Wouldn't it be cool if we could have code that would test itself?
> Yeah, if only there were some kind of automated way to encapsulate the requirements of classes and methods that could ensure that they behaved within some known boundaries...?
We've had automated unit tests for quite some time, but they are more often called compiler checks. Want your "automated unit tests" to be more comprehensive? Improve your type system and your compiler!
With all that said, I do not want to be dismissive of peoples' projects. This is fun and neat.
Joe Armstrong(of the Erlang fame), talks about this a bit here: https://youtu.be/TTM_b7EJg5E?t=778
I've pointed to a specific timestamp, but there might be more details somewhere else in that talk.
These are based on powerful types as "dependent types" which can encode mathematics and programs in the same programming language, and allow the mathematics to "talk about" the program. The mathematical underpinning is the Curry-Howard-Lambek correspondence.
[1] https://coq.inria.fr/ [2] https://github.com/mit-plv/fiat-crypto [3] https://compcert.org/ [4] https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
Too many people looking for silver bullets.
I would guess that Jupyter notebooks could easily be made into such a thing as well. Also doctests with editor integration might work like this already?