Launch HN: Rainforest QA (YC S12) – No-Code UI Test Automation
These startups were trying to push code through CI/CD pipelines as frequently as possible, but were stymied by quality assurance. Labor-intensive QA (specifically, smoke, regression, and other UI tests) tended to be the bottleneck preventing CI/CD from delivering on its promise of speed and efficiency. That left a frustrating dilemma for these teams: slow down the release to do QA, or move faster at the expense of product quality. Given that we were sure CI/CD would be the future of software development, we decided to dedicate our startup to solving this challenge.
For us, inspired at the time by Mechanical Turk, the question was: could we organize and train crowdsourced testers to do manual UI testing quickly, affordably, and accurately enough for CI/CD?
In the following years, we optimized crowd testing to be as fast as it could possibly be, including parallelization of work and 24/7, on-demand availability. (Our human-powered test suites complete in under 17 minutes, on average!) But, the fact is, for many rote tasks (like regression tests), humans will never be as fast or as affordable as the processing power of computers.
The logical conclusion is that teams should simply automate as much UI testing as possible. But we found that UI test automation is out of reach for many startups—it’s expensive to hire an engineer who has the skills to create and maintain such automated tests in one of the popular frameworks like Selenium. Worse, those tests tend to be brittle, further inflating maintenance costs.
With the rise of no-code, we saw an opportunity to make automated UI testing truly accessible to all companies and product contributors. So two years ago, we made a big decision to pivot the company and got to work building a no-code test automation framework from scratch. We’re excited to have launched our new platform this summer.
On our platform, anyone on your team can write, maintain, and run automated UI tests using a WYSIWYG test editor. Unlike other “no-code” test solutions which still require coding for test maintenance, our proprietary automation framework isn’t a front-end for Selenium. Unlike most test automation frameworks that test the DOM, our automation service interacts with and evaluates the UI of your app or website via machine-vision, to give you the confidence you’re testing exactly what your users and customers will experience. Minor, behind-the-scenes code changes that don’t affect the UI often break Selenium tests (i.e., create false positives), but not Rainforest tests.
Our automated tests return detailed results in under four minutes on average, providing regression steps, video recordings, and HTTP logs of every test. You don’t have to set up or pay extra for testing infrastructure, because it’s all included in the plans on our platform. Tests run on virtual machines in our cloud, including 40+ combinations of platforms and browsers. We build everything with CI/CD pipelines in mind, so most of our customers kick off tests using our API, CLI, or CircleCI.
Of course, not all tests can or should be automated; e.g. when a feature UI is changing frequently or when you need subjective feedback like, “Is this image clear?”. Today's computers are nowhere near able to replace the ingenuity and judgement of people; that’s why our crowd testing community isn’t going anywhere. But we can now say that Rainforest is the only QA platform that provides on-demand access to both no-code automated testing and manual testing by QA specialists.
We offer a free plan that provides five free hours of test automation every month, because we don’t think cost should make test automation inaccessible, either...
88 comments
[ 3.0 ms ] story [ 154 ms ] threadMaybe that’s what you’re looking for?
[0]: https://monitoro.co
I’d be curious to hear about the comparison to https://reflect.run — I can imagine that access to the tester community is a piece of that…
Outwardly, the way they automate is very similar to us. Looking a little deeper, it seems like they _do_ use the DOM pretty heavily (from re watching the video at https://reflect.run). For us, this is a fundamental difference - we do not believe in this; we want to automate testing-like-humans. It's harder, but we believe replicating how a human would detect things working or not (visually, via kvm) ends up with less brittle, easier to maintain tests that are closer to the reality of how a human would interact with your app.
Also, we test using VMs (or physical devices if needed for mobile) - allowing us to test the browser, or any other kind of software. This lets us support a large combination of OS and browser variants out of the box, or custom images for enterprise. Reflect doesn't seem to support more than Chrome when I last looked.
The workflow for creating tests in Reflect is pretty similar to Rainforest: we both expose a "cloud browser" that loads up your webapp and you interact with that to create your tests. The biggest difference workflow-wise is that Reflect records all your actions automatically, whereas with Rainforest you often need to both specify what step you're going to take, and then actually perform that action in the browser itself. Recording everything automatically is technically harder to pull off since it's forced us to ensure we accurately record every step you take, but we think it makes for a better workflow since you can create tests faster, and there's less chance of inaccuracies that cause tests to not be repeatable.
I would quibble with the statement that you need to be far more technical to use Reflect - we're a no-code product after all. :) We have plenty of folks who aren't developers using our product. But the good thing is that both products have free tiers, so users can always give us both a try for free and decide for themselves.
Edit: Also their statement about Reflect running in headless mode is incorrect. Our test grid is a cluster of VMs: we spin up a Docker container for each test run, and each Docker container is running the test steps using a normal non-headless browser.
And totally agreed, both products take a slightly different approach and have different strengths and weaknesses, try them both and see which is a better fit!
So, we went the route of having the same way of doing things for everything. We may change this in the future, but at the moment it's consistent and easy to learn. You add an action, then run it - no need to do it twice, or learn multiple ways of doing things.
How do you deal with things like permissions, proprietary information, etc?
Depending on what you mean; assuming it's the security angle:
TLDR; carefully
At a high level, most of our customers are testing in QA - not production - so usually the only proprietary information (outside of credentials to access it) we'd see is something they'd be releasing shortly anyway. However, we take security seriously;
Our infrastructure and code is heavily tested + reviewed before shipping, as well as externally audited yearly. Currently we're checked yearly for HIPAA, and from this have strong internal controls / processes, documentation and guidelines around access controls, how-things-are-done, and audited. Everything is encrypted at rest and transit (db, logs, images, etc). All the testing is done through our infra, recorded (video, kvm) and logged (http, https, dns, etc). Obviously we never re-use a VM, they're destroyed post use.
From the crowd side, they test using the same machines as automation uses (i.e. all the same logging levels as above). Additionally, each individual is KYC'd and signs an NDA with us before they can work. Enterprise, or folks needing BAAs have a sub-crowd with extra levels of KYC / other requirements.
We're currently early in starting being formally SOC 2, but it's not complete. More details here - https://go.rainforestqa.com/rs/601-CFF-493/images/Rainforest...
Launch HN: RescueTime (YC W08) – Redesigned for wellness, balance, remote work - https://news.ycombinator.com/item?id=28683597 - Sept 2021 (141 comments)
Context for everyone else: early 2014 we open-sourced https://github.com/rainforestapp/fourchette, which was an pioneer of Heroku Review Apps (https://devcenter.heroku.com/articles/github-integration-rev...), and generally the concept of doing this on pull requests over just per branch or commit.
The benefit of using code is that you have to know what you are doing. Having experienced what happens when people build data driven systems out of building blocks, with out a thorough understanding of what they are doing (brittle, failing is strange ways under load, general unreliability and low quality) I am suspicious
We remove the code requirement, making testing accessible to more folks - i.e. product managers and product designers who have great knowledge of the product, but don't want to or can't code. Also, this doesn't tend to exclude developers either. Currently our user base is roughly 1/3rd engineers, 1/3rd product managers/designers, and 1/3 QA folks.
The current code-based testing frameworks force me to add an unambiguous marker to the `input` element, like an attribute or ID, which also makes it easy to query from the DOM during the QA process. How does this QA product handle breaking changes to the UI, and how robust could you expect it to be to code changes?
Code is more upfront effort but more control and thus less fragility but more maintainable over time (if done right).
I imagine there are some/many situations where throwing many people at a problem is the “best” way and this would suit that quite well I guess.
Of course, until automation gets to be as clever of humans, any test automation approach is going to have some flavor of brittleness.
Do you a) run the tested software inside your VMs (if so, what's the integration API?) or b) expect your clients to run it (if so, how can the client authenticate your test access?)
a) we can, if so generally they install it as part of the testing (e.g. a client testing a chrome ext), or have us build a custom vm for them (e.g. clients with 20gb download)
b) this is the common path; folks push something, ci builds it, ships to a qa env, they run us, if it passes, push to prod.
For B, Auth is handled anywhere from zero auth (just fully open QA env, but usually it's SaaS so you still have to login to their app), through to http auth, limiting the IPs (https://help.rainforestqa.com/docs/which-ip-addresses-do-rai...), to VPN directly into their QA infra. Without pulling numbers, I'd guess 95% go the zero-auth route.
This is a constant dilemma for Solopreneurs/very small teams. I was just thinking last week - can I find an affordable automated testing platform (need to run tests on new features that I've added to my latest project - an Electron App).
Follow up question - does this work for Electron Apps? Either way, still happy to try it for other Web App projects if I do another Web App
This tool makes the benefits of a well-built automated testing setup much more accessible and less costly.
We had previously been using traditional automated QA tools like selenium when someone suggested Rainforest and I am very happy we made the switch. Nothing but praise and well wishes for this team - you are en route to massive success.
https://jobs.lever.co/rainforest
Congrats, you not only solve a big problem but introduce real advancement in UX with no-code option. Added value from real human QA is a wonderful bonus point.
Re the problem itself at a basic level, it is balancing visual matching (what does and doesn't matter in an image, what is the match in the VM - is that acceptable, and how/why?), and OCR (what is there, what matters, etc), and timing and interaction issues and complexity as well, and for us has to work on basically any platform we can run in KVM.