Launch HN: Reflect (YC S20) – No-code test automation for web apps

Thank you, there are two things that are either a bug, or hard to figure out.

1) I can't delete steps

2) I can't hover.

Great product guys, keep it up!

I'm using Hardware keys to secure my personal accounts. What are the alternative auth methods suggested for this situation?

I asked this elsewhere in the thread to a different user, but it seems relevant here as well:

Curious how your service would handle a app that mandates non-email based MFA like SMS or TOTP.

Additionally, what about testing a onboarding flow that might require some form of manual approval?

Thanks!

I recomment considering learning between test runs and I encourage you to train a relatively simple model for selection on top of http-archive and tagged data.

"off the shelf" machine learning makes it pretty easy to create very robust selectors. I gave a talk about it in GDG Israel and was supposed to speak about it in HalfStack that got delayed cancelled because of COVID19 - but the principle is pretty simple.

It's amazing how much RoI you can get from relatively simple models of reinforcement learning. Here are some ideas: https://docs.google.com/presentation/d/1OViIwDJJw1kjVJH5Z2N5...

Good luck :]

Fantastic! I like inglor's idea too.

Cool idea, thanks!

Nope, we were able to use their UI to create our tests just like any other user would -- everything we needed was supported out of the box (minus one minor workaround I had to add to my code because I had some janky, non-deterministic IDs on some dynamically added hidden fields).

Extremely biased of course :) But here's where I see the advantages of Reflect vs. Cypress:

- Cypress has a really nice local development experience that feels at home to front-end devs. I would describe Reflect as a nice remote recording experience that simulates an end-user. So we're kind of attacking the same problem from a different perspective. You can technically record Reflect tests against your local env with ngrok, but Cypress is certainly a better local testing experience. So advantage Cypress here.

- There are actions like interacting with iframes, drag and drop, links that open a new window, and file uploads that range from difficult to almost impossible to do in Cypress. We support these actions out-of-the-box.

- Similarly if you want to do visual testing in Cypress you'll need to integrate with a third-party visual testing tool like Percy or Applitools. We have visual testing support built-in.

- I've seen folks struggle bridging the gap between using Cypress for local testing and actually getting it set up in their build/deployment pipeline. Since we're a fully managed service it's just a single API call to get Reflect in your CI/CD. We also have built-in scheduling so if you just wanted to run your tests at a fixed schedule you don't need to integrate with anything, which I think is a nice way to ge t going quickly and prove out our tool.

- Because of Cypress's design, it's not so easy to get Cypress tests to run in parallel. This is really important because E2E tests take way longer to run vs. unit and integration tests, and the best lever IMO for reducing this is parallelization. This is also another tripping point for folks getting Cypress set up in CI/CD.

- The final differentiator I'll mention is really the difference between a code-based vs. codeless approach. We're trying to reduce the time it takes to create and maintain tests and we think the way to do that is to basically handle as much as we can automatically on your behalf. Instead of writing code that simulates actions, you just record your actions and we generate the automation for you. So for a flow like adding to cart or user registration, you might only take you a few minutes to set that up in Reflect but it'd be a lot longer to do in Cypress. Certainly as your test suite grows things like reusability become really important, and we support that as well. This also means that non-developers like QA testers w/o dev experience, PMs, designers, and customer support can write and maintain tests.

Had very similar experience with Cypress here as well. The command queuing API that abstracted away async operations sounded awesome in theory and I totally drank the coolaid.

But once I started to write tests with it, I realized the model was optimized for simple tests where you can afford to act only as the end-user, whereas for the more complex flows we're interested in testing, you need to act both as the end-user as well as, say, someone in the operations team doing a bunch of manual approval steps, or say, the email/sms system that delivering some secret out-of-band (through a staging backdoor API), interleaved between the end-user actions, and not being able to precisely coordinate those async interactions between these distinct actors made those tests extremely awkward to write compared to something using playwright/puppeteer, where you just coordinate promises like you usually do.

That's before we get into the limitations around single domain per test which meant we couldn't effectively test our login flow that uses a separate domain for login for OIDC.

That's not the kind of versioning I intended. What I'm interested in is if I make some changes to the test setup in your UI, is there a way to see what I changed, and when those changes happened in the past?

One thing that will come up and we've found out: users want branches, they want merging and they want the ability to track their git branches with their tests.

Yea I don't want to discourage work in the space. Actually, quite the opposite. I'm just saying where the bar is for me when I'm considering handing over money. I'm not going to pay money for something that ends up actually slowing me down.

How does that address the issue of the UI changing? If the markup changed, you still need to update your tests.

We tend to create a "view model" object which maintains the state of the UI and interacts with the server. The actual view just forwards commands to this object, and after a command happens the view always re-renders based on the current state of the model. We do this on iOS but I've been pushing for us to do it with React as well since you get the built-in state rendering for free.

So all of the tests look something like this (here's a UI form):

  // Arrange
  let searchResponse = { usernameTaken: false };
  let networkService = NetworkServiceTestDouble(searchResponse);
  let vm = UserFormViewModel(
    networkService: networkService
  );

  // Act
  vm.username = "newusername@email.com";
  vm.checkUsernameAvailable();

  vm.submitForm();

  // Assert
  expect(networkService.calls[0].arguments[0]).to.eq({
    username: "newusername@email.com"
  });

So you see the view model is kind of like a Page Object, in that it has an API which represents the user's interaction. But, in this case the view model is what's used in the actual production code, it's not just a testing helper. So you just invoke the API as the user would when interacting with the feature, and verify that the correct data gets sent to the server when the form is submitted. After all, the client's job is primarily to interact with the server and maintain state. The feature is functional if the correct data is sent to the server.

Then, let's say you're in React, hooking up this view model is quite easy:

  import { UserFormViewModel } from "./UserFormViewModel";

  let vm = UserFormViewModel(realNetworkService);

  function UserForm() {
    return (
      <button onClick={vm.submitForm()}>Submit</button>
    )
  }

The buttons / inputs just get wired up to the appropriate view model functions and properties, and you can have the component's state just be the view model's state. The view is then just entirely about appearance and forwarding events.

I'll then write some tests that actually render the UI, but they're just sanity checks. How often do you get a bug where you failed to call the correct function in response to a button click? The logic and wiring in response to all of the events is the more interesting part.

Edited to improve the code formatting a bit.

Thanks for answering those questions.

I think it's all great news, and for customers it's reassuring to know that running time is really just about the test itself (which I noticed is in the FAQ, so sorry for wasting your time there). This makes me think about a couple more things: can clients select zones from which to run the tests? It's probably not a huge focus if your current target customers are in North America, but can become useful if a European company considers the tool and all loading times become noticeable. Or just if a company expects clients from anywhere and wants to test for that (which is basically the hardest bit for a small team, as it starts to involve having some devops experience to spin up VMs on the cloud in different zones and all that jazz). Another tangentially related setting would be the locale for tests execution and scenarios (I'm thinking of date formats, monetary symbols, etc.. with expectations being different if you're in the US or in the UK for example)

The lack of cross-browser testing is something I can see scaring away some teams, especially with mobile being restricted to profiles of older iPhones only. IIRC over 70-80% of iPhone users are on the latest major iOS version. But it all ties in to other issues that I'm sure you're already dealing with (iPhone users using Safari, something like 20% of desktop users using Firefox/Safari, and the odd clients here and there who would have very specific annoying requirement like "I must be able to test on Internet Explorer - yes I know it's deprecated but my startup is B2B and I'm the small fish there").

Your docs might be short but they are already good and readable, especially for a target that might include a mix of pure devs and automation QAs/SDET/etc.

All the best!