Launch HN: Meticulous (YC S21) – Catch JavaScript errors before they hit prod
I was inspired to start Meticulous from my time at Dropbox, where we had regular 'bug bashes' for our UX. Five or six engineers would go to a meeting room and click through different flows to try to break what we built. These were effective but time consuming—they required us to click through the same set of actions each time prior to a release.
This prompted me to start thinking about replaying sessions to automatically catch regressions. You can't replay against production since you might mutate production data or cause side effects. You could replay against staging, but a lot of companies don't have a staging environment that is representative of production. In addition, you need a mechanism to reset state after each replayed session (imagine replaying a user signing up to your web application).
We designed Meticulous with a focus on regressions, which I think are a particularly painful class of bug. They tend to occur in flows which users are actively using, and the number of regressions generally scales with the size and complexity of a codebase, which tends to always increase.
You can use Meticulous on any website, not just your own. For example, you can start recording a session, then go sign up to (say) amazon.com, then create a simple test which consists of replaying against amazon.com twice and comparing the resulting screenshots. You can also watch recordings and replays on the Meticulous dashboard. Of course, normally you would replay against the base commit and head commit of a PR, as opposed to the production site twice.
Our API is currently quite low-level. The Meticulous CLI allows you to do three things:
1) You can use 'yarn meticulous record' to open a browser which you can then use to record a session on a URL of your choice, like localhost. You can also inject our JS snippet onto staging, local, dev and QA environments if you want to capture a larger pool of sessions. This is intended for testing your own stuff! If you inject our snippet, please ask for the consent of your colleagues before recording their workflows. I would advise against production deployments, because our redaction is currently very basic.
2) You can use 'yarn meticulous replay' to replay a session against a URL of your choice. During replay, we spin up a browser and simulate click events with Puppeteer. A list of exceptions and network logs are written to disk. A screenshot is taken at the end of the replay and written to disk.
3) You can use 'yarn meticulous screenshot-diff' to diff two screenshots.
There are lots of potential use cases here. You could build a system on top of the screenshot diffing to detect major regressions with a UX flow. You could also try to diff exceptions encountered during replay to detect new uncaught JS exceptions. We plan to build a higher-level product which will provide some testing out of the box.
Meticulous captures network traffic at record-time and mocks out network calls at replay-time. This isolates the frontend and avoids causing any side effects. However, this approach does have a few problems. The first is that you can't test backend changes or integration changes, only frontend changes. (We are going to make network-stubbing optional, though, so that you can replay against a staging environment if you wish.) The second problem with our approach is that if your API significantly changes, you will need to record a new set of sessions to test against. A third problem is that we don't yet support web applications which rely heavily upon server-side rendering. However, we felt these trade-offs were worth it to make Meticulous agnos...
41 comments
[ 5.7 ms ] story [ 95.4 ms ] threadLooks very promising, wish the team the best!
Catch those errors before hitting prod, sounds like the dream
PS: As on open sourcing the record+replay code, I'm sure that'd be awesome, I only have this on my radar https://github.com/openreplay/openreplay as a FOSS alternative to fullstory/logrocket for now.
> Open sourcing
Openreplay is awesome, but we ended up building heavily on top of rrweb (https://github.com/rrweb-io/rrweb). Did you know they have their own documentary on the project? I only noticed that today.
Performance is probably the area I've spent the most time thinking about: if you want to measure performance regressions in a page, instrumenting it with a session recorder is definitely a way to skew the results (for example, checking scroll position of elements during snapshotting will trigger a reflow).
With waldo.io and Checkly It joins the list of QA force multipliers that would make my life, as the sole developer in a bootstrapped startup trying to punch above it's weight, much easier. First, they give me a taste with a free plan, then hit me with production pricing we still can't justify.
If I have this right, 20 sessions is just a trial and 1000 sessions is very careful use.
If these scenarios are so easy to create, I would imagine you would make something like 50 (?), run them against every deploy (10 a day?). That's 15.000 replays, right?
So the 100$ plan is something like 10 scenarios at 3 deploys a day. That sounds too scaled back to get good use out of it. Or do I have the wrong idea about the intended use case?
I get it though: they all have reasonable pricing for a unique service that provides real, obvious value AND may actually be expensive to run. I'm just a little sad about not getting to use them.
Biggest concern I would have is portability. One benefit of testing suites, when done right, is they gain more coverage over time, especially against regression bugs. I would be very concerned about building up a large suite of tests for my most critical flows on proprietary tech that could be rendered worthless in an instant if the company goes bust, decides to pivot, etc.
Overall the idea works very well and Testim had/has a lot of customers using record/playback.
> Portability of the data.
I hear your concern here. The replay data and session data is also saved to disk, so you can save this somewhere. Of course that still leaves the risk of the record & replay tech. I think open sourcing this would solve the portability issue here, and it's something we're actively talking about but haven't reached a conclusion on yet. Anecdotes and examples like this though are incredibly helpful in helping us make that decision.
Thank you for the feedback!
You should have used waits instead of sleep. Wait pools until your DOM is in the state you are waiting it to be, unless given timeout occurs: https://www.selenium.dev/documentation/webdriver/waits/#expl...
Especially when system has to warm up.
In the end it is Okay
We tried and failed to create a “bug capture” offering in Testim.io - what helped us work with comapnies like Microsoft/Salesforce and eventually make an exit and sell to a much larger player (Tricentis) is focusing on rock-solid AI improving tests. The founder still believes the capture idea (qa capture bugs for devs) has a lot of merit but I think there are fundamental issues with anything that doesn’t reproduce timing perfectly (some do like Firefox’s replay.
I’m not with Testim anymore but still very excited people tacking this problem and I warmly recommend pinging Oren@testim.io (the founder, an engineer, a GDE and a nice guy) for pointers - he likes giving free advice and investing in new players in the space to cultivate the ecosystem (most companies currently have no e2e tests)
I would be curious what percentage of corporate repos have any tests.
> Testim/QA Capture
Self-improving tests is a really interesting area. Timing is definitely an intricate issue. You probably have to layer on top of each other a bunch of different and novel techniques to get something with good signal-to-noise. We're still working on developing those out :)
Oooh, thank you for the rec. I'll make sure to ping Oren after the launch. The space is enormous and my understanding is that the rate of growth for testing tooling will exceed the rate of growth for software, which leaves QA and testing companies in a good position.
Meticulous captures network traffic at record-time and stubs in responses at replay-time, which removes the need for a backend environment to replay against. We also apply a little fuzziness in our replay, like how we pick and choose selectors (e.g. imagine css selectors with hashes in them, the 'same selector' will look very different between two builds).
We have a long way to go in making this robust though.
Is there anything you wish was easier when writing tests with Playwright?
That's neat.
We've only just started using Playwright, but we're surprised at how easy it is to get going; but we're also all developers. We primarily use it to test large feature flows that are hard to mock in unit tests. For example, one test logs you in, uploads a file, waits for the result, clicks on the download button and makes sure the downloaded file is what we expected. We mainly want to ensure that we don't accidentally delete or hide the login widget or the download button while working on something tangentially related.
In the example outlined above, we don't mind spinning up the backend locally, as this allows the test also to make sure the response is correct.
However, I see how being able to only test front-end code quickly and easily without the need for a backend can be helpful in many applications. Congrats on the launch, and good luck!
I've been looking forward to this launch for a while; I've spent a lot of time experimenting with session recording and how it can work for regression testing, reproducing bugs, measuring performance, sharing feedback during development - there is so much potential here.
> We're actively discussing open sourcing the record+replay code.
I think open source is a good call - supporting an option to self-host would make a lot of sense, since session recording will inevitably slurp up PII or sensitive data which could put off some users.
Gabriel and the team are really awesome and this product is a genius idea - I'll definitely be using it at my company as it could save us a ton of work in setting up our testing pipelines.
- Alejandro :)
With regards to updating existing records, unfortunately we don't currently have good tooling & support for this, so you may need to record new sets of sessions as your application changes. I would suggest starting off with testing a few core flows.
This reminds me of the test suite for one of my projects where E2E tests only covered relatively simple scenarios because of subpar WS mocking support, leaving more important, complex interactions to manual (can't run in CI) or fully integrated (expensive to author and run often) testing. The situation changed only after we wrote a custom WS mocking layer over the HTTP mocking the framework provides, yielding a dramatic increase in coverage. Out of dozens of developers I interviewed, only a few solved this issue to some degree. Clearly, mainstream testing frameworks provide insufficient support for the use case.
But I agree that it would enable replay for a lot of very interesting, complex projects. I've worked on some FX trading UIs that have been challenging to test without standing up a lot of backend services.
Reading through thus just reminded me of Datadog browser tests. It's not exactly the same, but it might be interesting to check them out.
If you’re able to spin up an environment via docker-compose and play against that with playwright, then I think that’s good for that use-case.
However, if you’re testing a flow that relies on some initial state, it can sometimes be tricky to seed that state or do so in a way which is representative.
I think software development is due for a disruption and your take on testing is spot on.
As part of a dev tool belt we developing, we are building a tool to translate user interaction into a selenium script, and have the selenium script run on our server. User get to take away the script so they don't get vendor lock-in. What is your approach into replaying a user session?
On a broader picture, I think what you do has potential beyond QA. e.g. if run on production, have CS hop on the same session as a troubled user.
Just checked your profile. It seems we both are based in london. Maybe we can grab a beer to discuss the potential.
jQuery, Angular, React
are analogous to:
Selenium, Puppeteer, Playwright
Is Selenium still worth considering for a brand new project, though primarily due to ecosystem instead of implementation (Ruby, IDE, etc.)?
To frame the question in context of the analogy, though now completely off-topic:
What is the React equivalent of datatables.net?
How do you deal with generating resilient selectors automatically? That's a class of problem which plagues this type of tool in my understanding.
My other concern is I would want it to be open source and self hostable. That runs against being commercial though but serious companies will pay a licence for support. If not then a solid promise that the company phoenixes to open source if it shuts down, and allows closed source self hosting while open.