HN Explorer

Show HN: Lost Pixel – open-source visual regression testing for your frontend (github.com)

90 points by divdev_ ↗ HN
Hey! My name is Dima, I am from Ukraine and I am the co-founder of Lost Pixel! We built this tool to solve our problems at work and decided to open-source it so more people can build their custom visual regression testing pipelines! If you want to chat about the tool or visual regression testing in general I am super excited to meet like-minded people! Thanks a lot for checking out the product and I hope it will serve you well if you decide to try it out :D

15 comments

[ 1.6 ms ] story [ 45.7 ms ] thread
[deleted]
(comment deleted)
kilian
I've set up visreg tests in the past and they were always slow and brittle. This looks really modern and well documented system, congrats to the team! Always running them on GitHub actions will keep the sub pixel issues at a minimum, but do you also have configurable thresholds?
divdev_
Hey! Thanks for the nice words, yes we do - thresholds is indispensable part of Lost Pixel, after all we don’t want to battle the tests which are 2-3 pixels different, those should pass by default.
Akcium
I think I have already seen your project on Twitter, remember the nice gif.

Wish you luck with the project.

mattlondon
How do you handle "meaningless" visual changes? E.g. chrome version change introduces super-subtle blending changes to drop-shadows, and suddenly there are huge diffs because the least significant bit of a drop-shadow's grey colour changed. What is your approach to that?

So far all tools I have used like this suffer from "false positives" like this (it's a true positive technically of course, but likely false from an engineering perspective).

divdev_
We don’t at the moment differentiate, but it is definitely a feature on the roadmap. There are multiple ways to solve this and we are considering quite some of them:

1. Grayscaling browser before screenshot 2. Using custom diffing engines that enable for ignoring regions like odiff 3. Trying to be smart and analyzing images with machine learning to identify how meaningful the difference is (moonshot but worth exploring)

Our approach to this in general is that if false positive like this happens we can just batch approve all the differences and be ok with that as it was a subtle rendering change not dependent on us or our direct third party dependencies.

testingwaters4
You could run the suite on the updated browser on the latest passed build, then automatically batch approve since you know it has to be green. Now you can run all the newest test suites in this updated browser against this approved build.
divdev_
exactly this is the current workflow for many people who are using visual regression tests!
kasperg
The open-source angle aside: How does Lost Pixel compare to Chromatic?

It might be nice to add a comparison to your otherwise excellent documentation.

divdev_
We are not backed by any VC - just indie hacking that's the greatest prerequisite to comparing ourselves to Chromatic. At the SaaS level we want to bring value to the community by:

1. allowing visual regression testing integration in any part of your flow(locally for free, paid on the platform) with any need for visual testing(stories, pages, components, cypress, playwright). Chromatic is focused on building upon their fundament - storybook. 2. making it affordable. Everybody agrees to one thing - the prices to visual tests are bonkers when it comes to SaaS. We embraced the fact that people want flexible and cheap as much as they want fast and expensive. We will allow to run actual engine on your premises making our pricing significantly cheaper, we will allow storing images on your side which will again make it ridiculously cheap for you as you won't be paying markup that we otherwise could induce on those things. 3. not being protective of our codebase for core part of the offer. We want community to build the top-notch visual regression testing engine with us to serve the community!

Hope it sheds some light into bigger picture, otherwise Chris & I will definitely add the table for comparison with Chromatic & Percy. this is very legit point, thank you!

acketon
Hi Dima, this looks really good! I see examples for Storybook and Next.js. Would this work with other technologies and configurations? For instance is there a way to point it at an external dev sandbox to capture full page screenshots of an existing site... say a WordPress site or other CMS? I have a use case where I want to test full page screenshots but need the environment that is hosting a copy of the content/database to test against, not just the components used in the pages.
divdev_
hey! Absolutely, Lost Pixel is designed to be able to test anything which can be opened in a browser window. So any technology and configuration should be quite flexible to be tested as long as the host is publicly accessible from your runner of lost-pixel engine. So in your case if you can run your environment somewhere and point lost-pixel there it should work as expected! Feel free to reach out if you need help - we are happy to chat and assist you! Github discussions or @divdev_ or @chriskalmar on twitter are both good ways to reach us!
fishtoaster
This is very cool! I've found visual regression testing in CI to be one of the highest-value things I can throw into new, long-lived frontend codebase. I've been relying on Chromatic so far, but I'd definitely try out Lost Pixel next time I'm setting up such a project!
divdev_
Visual regression tests rock! Be sure to ping me (@divdev_ on twitter) when you will be doing that - would be very eager to pair with you on this and help you set everything up!