Show HN: Checksum – generate and maintain end-to-end tests using AI
I’m Gal, co-founder at Checksum (https://checksum.ai). Checksum is a tool for automatically generating and maintaining end-to-end tests using AI.
I cut my teeth in applied ML in 2016 at a maritime tech company called TSG, based in Israel. When I was there, I worked on a cool product that used machine learning to detect suspicious vehicles. Radar data is pretty tough for humans to parse, but a great fit for AI – and it worked very well for detecting smugglers, terrorist activity, and that sort of thing.
In 2021, after a few years working in big tech (Lyft, Google), I joined a YC company, seer W21, as CTO. This is where I experienced the unique pain of trying to keep end-to-end tests in a good state. The app was quite featureful, and it was a struggle to get and maintain good test coverage.
Like the suspicious maritime vehicle problem I had previously encountered, building and maintaining E2E tests had all the markings of a problem where machines could outperform humans. Also, in the early user interviews, it became clear that this problem wasn’t one that just went away as organizations grew past the startup phase, but one that got even more tangled up and unpleasant.
We’ve been building the product for a little over a year now, and it’s been interesting to learn that some problems were surprisingly easy, and others unusually tough. To get the data we need to train our models, we use the same underlying technology that tools like Fullstory and Hotjar use, and it works quite well. Also, we’re able to get good tests from relatively few user sessions (in most cases, fewer than 200 sessions).
Right now, the models are really good at improving test coverage for featureful web-apps that don’t have much coverage (ie; generating and maintaining a bunch of new tests), but making existing tests better has been a tougher nut to crack. We don’t have as much of a place in organizations where test coverage is great and test quality is medium-to-poor, but we’re keen to develop in that direction.
We’re still early, and spend basically all of our time working with a small handful of design partners (mostly medium-sized startups struggling with test coverage), but it felt like time to share with the HN community.
Thanks so much, happy to answer any questions, and excited to hear your thoughts!
33 comments
[ 3.8 ms ] story [ 81.4 ms ] threadHow well do the tests hold up over time, and how well are the tests validating the contract of the code instead of just historical behavior and quirks?
We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible.
We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.
Then, during test generation, we bias the model to explore edge cases (in a few ways), and the model is still able to complete those even with low sample.
In other words, we direct the model toward certain goals, and flows and also add chaos to the process which result in the model executing unexpected flows.
Thinking of apps that might fall under HIPAA etc
1. We hash all inner text and then backfill static strings on the server side. So every text that is specific to the user remains hashed
2. We detect special cases like passwords, SSNs, credit cards, and completely block it (even not hashed)
3. We provide full privacy controls to our customers to easily mask any sensitive elements
4. We discard the user IP and don't require any PII to be sent. So we can connect a session together, but don't really know who the user is
> How is the product different from the other test generation tools
We train our models based on real user sessions. So our tests are: 1. Completely auto-generated 2. Achieve high coverage of real user flows, including detecting edge cases 3. Automatically maintained and execute with our models so they are less flakey.
> How do you check if the are testing the intended behavior
Our models are trained on many real sessions so it learns how your website (and others) should behave. In that sense, it's similar to a manual QA tester which can detect bugs. To supplement for functionality that is not obvious by the UI, we are now looking at adding LLMs to parse code, but most of the functionality can be inferred from the UI
How can it know, by watching my clicks, how I decide if the behavior is correct on the backend?
But the tl;dr is 1. We learn how to use your app based on real sessions (we remove sensitive information on the client side) 2. We train a model on this data 3. We connect this model to a browser and generate Playwright or Cypress tests
The end result is code written and Playwright or Cypress. You can edit and run the tests regularly
Feel free to sign up for a demo if that's a priority for your team. Even if it's just to chat and connect.
There's nothing "specific" in the underlying model that prevents it from testing mobile. It's just a matter of focus at the current time.
> Our impact on pefromence is non-existant as we use battle-tested open source tools used by Fortune 500 companies
Is it for the identification of user sessions that are good candidates to make into tests? Is it the generation of test specification in some DSL / Cucumber / Selenium / etc.?
1. We use AI to analyze the user patterns and find common paths and edge cases, basically building a representation of your UX in a DB
2. We then use the DB to train another ML model that learns how to use your app the same way a user does. Given a certain page and user context, the ML can complete UX flows.
3. Finally, we learn to generate assertions, run the tests and convert the model actions in step 2 into proper Playwright or Cypress tests
What does that mean, exactly? Just because it's open source and used by F500s doesn't mean it can't have performance issues.
1. We (and others) have tested our tools' impact on memory, CPU, network performance and found only negligible impact, even on slower/older devices
2. Also, they are used by F500 companies and have wide adoption, which indicates that other well established devs have run the same tests and decided to move forward.
We'll work on the language there to clarify.
-- I think the name doesn't sell me or even most people because "checksum" is more of a security/crypto term. When I saw the HN post say Checksum I didn't think it was going to be about end-to-end tests. I thought it was going to be some crypto thing. Maybe a name like "Tested" or "Covered" is going to click better with the potential customer.
-- I don't feel like the demo video is making me feel like I know what this product is doing. I could also be misunderstanding the product. It might help more if the demo showed the following (in ideally less than 5-10s or most users might tune out):
1. A quick setup step for checksum 2. A set of generated tests 3. Passing tests
Seeing those steps would give me the emotion as an end-user "wow this must something I can quickly setup and will make me feel like I have test coverage out of the box"
We don’t have pricing, not because we try to be vague, but because we haven’t fully figured out our training costs, which can vary significantly per app. We are very much in the “Do things that don’t scale” phase where we hand-pick our customers, provide white-glove treatment and prioritize learnings over price
But as a tester, I would like to see a tool that isn’t just more bullshit. For this to happen you will have to explain:
- What exactly is your product designed to do? What kind of products can it be applied to test?
- What do you mean by the word test? Humans test in many ways and levels. Do you simply mean “exercise code while detecting crashes?” Because that’s a tiny part of testing.
- Code coverage is not the only kind of coverage. So how do you automatically achieve state and data coverage? I’m guessing you don’t, but hoping you will surprise me.
- Test oracles come in all shapes and sizes. One of the reasons I say testing cannot be automated is that I can easily demonstrate that a human tester cannot fully specify their own oracles, and thus cannot write code to implement them, either. So, how does your product recognize a bug when it sees it?
- How much human handholding is needed to operate your product?
- Testers think critically about how users interact with the product as users attempt to fulfill their purposes. This guides practical testing. I haven’t yet seen any product that thinks critically. ChatGPT can’t. So how does your product cope?
- When the product under test changes, what does your product do?
- Can your product EXPLAIN its test coverage (other than reporting code coverage, which is a poor indicator of good test coverage)?
- Say I have a product that sends the user through a multimodal questionnaire (including the use of animated screens that guide the user through measuring heartrate) and then produces a diagnosis of possible illnesses. Can your product tell if the diagnosis was correct in relation to the original intent of the logic that is documented in Jira tickets and Slack conversations? Will it generate questions about any of that, the way a real tester does?
0 - seriously rethink your branding. I can help.
1 - how does the Ai know when the test is successful? — is it a visual comparison? — if so.. is there a threshold range that can be adjusted?
2 - how does this differ from https://www.meticulous.ai/
3 - would it work on highly complex UX/Ui interactions like these here?
https://youtu.be/WtglzRWQzVE