HN Explorer

Show HN: Touca – a better alternative to snapshot testing (github.com)

63 points by pejman_gh ↗ HN
Hi everyone,

Almost 2 years ago, I left my full-time job at Canon to build tooling and infrastructure to help developers write high-level tests for complex software workflows that are not easy to unit test. I wanted to take ideas from visual regression testing, snapshot testing, and property-based testing and build a general-purpose regression testing system that developers can use to find the unintended side-effects of their day-to-day code changes during the development stage.

After two years of working ~70 hours per week and going through multiple iterations, we finally have a fully open-source (Apache-2.0) product that finally makes me and other members of our community happy: https://github.com/trytouca/trytouca

This week we released v2.0, a milestone version that is useful to small and large teams alike. This version comes with:

- An easy to self-host server that stores test results for new versions of your software workflows, automatically compares them against a previous baseline version, and reports any differences in behavior or performance.

- A CLI that enables snapshot testing without using snapshot files. It lets you capture the actual output of your software and remotely compare it against a previous version without having to write code or to locally store the previous output.

- 4 SDKs in Python, C++, Java, JavaScript that let you write high-level tests to capture values of variables and runtime of functions for different test cases and submit them to the Touca server.

- Test runner and GitHub action plugins that help you continuously run your tests as part of the CI and find breaking changes before merging PRs.

I would really appreciate your honest feedback, positive or negative, about Touca. Would love to learn if you find this useful and look forward to hearing your thoughts and answering any questions.

15 comments

[ 4.2 ms ] story [ 43.5 ms ] thread
crgwbr
Isn’t a lack of snapshot files a downside? How do you review snapshot differences tied to a merge request? Being able to review code changes right along side the test changes and snapshot differences is a big plus for normal snapshot files in my book.
pejman_gh
Hi there, maker of Touca here. Thanks for sharing your thoughts! Being able to visualize the differences along side pull requests is helpful. We currently provide that insight for each commit using Github Actions job summary and plan to add a GitHub App in the future to report differences as a PR comment. Touca also allows locally storing the captured data points for users who like to do so . We argue that visualizing the differences shouldn't require storing snapshot files in version control because 1. they are inconvenient to maintain 2. what you can store in them is limited to what your application produces as output. I've written a bit about this in our docs here: https://touca.io/docs/guides/vs-snapshot/ Would really appreciate hearing if you find this argument reasonable.
crgwbr
I’d argue that integration into GitHub via actions isn’t good enough—what about being able to review diffs using the git CLI? Or investigating snapshots with git blame?
pejman_gh
I agree GitHub Actions job summary does not take the place of a GitHub App. The latter is part of our Q1 roadmap.

Reviewing diffs using `git diff` or `git blame` is not part of our roadmap. In fact we want to replace that model so that we can provide more insights and better summaries than just showing the diff between two textual outputs. Currently, you can use the Touca CLI to capture test results and get feedback whether they are any different. But you'd need to use our web interface to see and manage the comparison results in details. We're working to expand the CLI so that you could see the diff between any two versions right in the terminal.

andrewfong
Similar concern here about the lack of snapshot files. Can "locally stored" captured data points be version controlled, even if it's some special Touca binary that makes no sense in GitHub?

I'm coming at this from a "how would I integrate this into my existing corporate setup" perspective. I'm concerned less about UX and more just how it works.

If I interact with Touca via a local client that doesn't to the Internet, and the actual data points are exchanged entirely via Git, that's a much easier sell than another server that I have to ask SREs to maintain. And being able to say "it's just Git + a local client" makes it easier to reason about concerns like "what if we switch to GitLab" or "how would this integrate into our own bespoke CI/deploy pipeline".

pejman_gh
Hi Andrew, to clarify, we do support generating local snapshot files in either JSON or Binary with FlatBuffers format (see https://touca.io/docs/sdk/capturing/#locally-storing-test-re...). We also provide `touca results` which helps with managing binary result files (see https://touca.io/docs/cli/results/). So it is possible to forgo the Touca server and just use the SDKs to capture values of interesting variables or runtime of important functions.

Storing test results in Git and comparing them via external tools may make it easier for teams to get started with Touca. But we care a lot about the developer experience and think that having a remote server take care of storage, comparison, visualization, and reporting could make a lot of sense for teams building serious software.

Touca has several components that work great together but you can certainly pick and choose and don't have to use all of them. We like Touca to do as much work as possible including integration with CI so that tests run continuously. But that integration is also optional. If teams have a different CI than GitHub Actions, they could call `touca test` as a CI step instead of using our plugins.

Does my explanation address your concern?

drothlis
It side-steps the problem of git conflicts, I suppose. You'd have to use their tool (`touca diff`? I don't know if that exists) instead of `git diff`.
relbeek2
Testing is a big part of our CI/CD pipeline, if we are making changes we should definitely see a diff in some functions.

I would expect to have a build fail on unintended changes like we do with snapshots. How exactly do you indicate intended changes in Touca?

pejman_gh
Unless a special command-line flag is provided, the test runners return a success exit status. Since we don't know whether potential differences are intended, we don't want to fail the CI by default. When differences in a given version are expected, you can promote that version as the new baseline through the web interface. This way, the test results for all subsequent versions will automatically be compared against the new version.
ajjenkins
I like this. TDD purists would probably complain that you don’t assert a specific value, but in practice I know a lot of people hardcode the assert values for their tests based on the test output. So this just automated that, I guess. If I was looking for a new test framework, I would consider this.
pejman_gh
Glad to hear. Yes, your understanding is very accurate. TDD requires a different mindset. Our primary focus has been do a great job of finding side effects of code changes to existing systems such as day-to-day refactors and occasional rewrites. From my personal experience, these are the scenarios in which having hard-coded expected values in assertions is inconvenient and requires mechanical adjustments. Thanks for taking the time to review this and sharing your thoughts.
drothlis
Some ideas I got from Jeremias Rõßler's talk: https://t.co/xWtA58Q9q5

- Snapshot testing is like version-control but for the outputs rather than the inputs (source code).

- Asserts in traditional unit tests are like "block lists" specifying which changes aren't allowed. Instead, snapshot testing allows you to specify an "allow list" of acceptable differences (e.g. timestamps).

spuz
In the example in the README it says "Touca tests do not hard-code input values" and yet it hardcodes inputs of the find_students function to "Alice", "Bob" and "Charlie". Is this a mistake? How does touca produce inputs if they are not hardcoded?
pejman_gh
Hello! Sorry for the confusion. What we mean is that, unlike unit testing, the test inputs are not tightly coupled with the test code and you can write a single block of test code and have it executed for any number of inputs.

The example in Readme is passing the inputs as a list. You could also use a lambda function (see https://touca.io/docs/sdk/testcases/#programmatic-testcase-d...). Alternatively, you can remove the list and pass the inputs as command-line arguments. You can even choose not to pass them at all, in which case the test runner will re-use the list of test cases in the baseline version.

goncalo-r
Looks like a really useful tool. A lot of codebases I worked in waste a lot of effort adding tests that just try to capture some golden expected values. As these aren't necessarily tied to product requirements, they often change and require devs to change the tests to match any new output. I can see a tool like this greatly reducing the time spent on these tests while providing better coverage and detecting more regressions.