I have some tweaks i was going to merge in that gave the diff's as %'s (so you could ignore small differences like fonts being different) as well as animate between the two versions. On my TODO was grabbing the actual element style attributes and then actually just diffing that, so the differences would be more or less spelled out.
This looks promising. Our project has a plethora of selenium tests, but they're focused on functionality only, not appearance. We can tell there is a table, button, link, form, etc. and click it. On that side we're reliable.
But we really let ourselves down sometimes with regressions to the look and feel of the application.
Are there any competitor tools for this visual testing ?
It's written in Ruby but is not really dependent on Rails or anything to run. We use it to compare a list of URLs between the version we are currently testing in CircleCI and our production site. Any differences are highlighted in a nice gallery and we get a message in our Hipchat room.
You can try https://app.webmate.io. It's a hosted service for layout, regression, and cross-browser testing. Basic features are free. The diff-ing works on the DOM, is visualised in screenshots and identifies the XPath of elements that are found to be changed, removed, or added. Tests are run on browsers/vm that are part of the service, giving you access to 200+ os/browser combinations.
Actually it's all home grown, right now it renders the page and screenshots in Firefox 38 ESR and then the actual diff image is produced by imagemagick's compare function, with some special things around it to handle edge cases with width/height differences and some others.
This is neat. Recently started doing Selenium tests with Scrutinizer CI, but we've had issues where the UI is broken and we find out a day later... I'll definitely keep an eye out for you guys
Are you an one man army? Congrats! :) This is great. How do you plan to react to potential changes in rendering engines? Would those be shown as test failures as well?
Great question! And yes. :) My plan at the moment for that is, because I know which browser exactly it was rendered in originally, I can put a notice at the top of the page to say something like "We just upgraded our rendering browser! The diffs below might because of that and not because of a change you made." -- and then after that one build, things would be back to normal.
If you are interested in how something like this was made, there was a good talk about building something like this at RailsConf this year: http://youtu.be/Q4ttqkIEM7g
Percy.io's Invite Request system is tied to github.
How is that not a problem imposed by Percy?
I'd feel more comfortable having a different mechanism to request an invite.
While the permissions don't grant Code Access, it does ask to read Organization and Team membership. I'd expose data for every member of my companies github organization to Percy, JUST for an invite.
This seems like a really convenient way to watch for front-end changes, but how do things like margins work? Will my whole page be red because everything has been shifted around or squished?
Responsive capabilities will really improve usefulness, and the absence is somewhat of a deal-breaker. On responsive sites developers will have to eyeball each change even when using Percy, so checking the desktop resolution wouldn't be much additional effort relative to checking all other resolutions for each page.
On the biz-dev side, if it's going to be a paid product it might be a good idea to start charging ASAP!
It has been integrated with Travis CI and GitHub pull requests so that non-passing diffs are appended as a comment to the pull request by a bot account.
If there's an intended diff, a pull request can also automatically be submitted against the original pull request which, on merge, updates the "gold master" images and allows the original pull request to pass.
This looks really great. The integration with Github and CircleCI makes it a very attractive option for us as it would slot right in to our existing workflows.
I think they might only do live sites, the thing about Percy is that it can be integrated earlier into your feature/browser tests for all pages, even those behind logins.
I've been working on VisualReview (https://github.com/xebia/VisualReview), an open source project for visually reviewing a (web-)application as part of a pull-request, very similar to percy.io.
We're releasing our first version somewhere next week which includes a basic comparison GUI and testing api's. PR workflow planned in the release after that. Interesting to see we're not the only ones with this idea.
I signed up. Clicked on the survey, clicked back, got a blank page. Did it again, got a live page. Did it one final time, got a blank page. Tried it again just now, blank page (but there is HTML in the page source).
It depends on the project. But when you're dealing with mobile devices you don't get to test even a fraction of the models available unless you've got enough scale to have a dedicated testing lab.
Hey! Good question, we're still working out the details on pricing but I want it to be very reasonable, simple and transparent for people. Just need some time to work through with a few initial beta testers. Stay tuned!
Check out also http://shoov.io/ - there are a few blog posts, that explain the concepts and point to examples. Apart of the visual regression testing it also allows monitoring the live sites with your preferred functional testing frameworks (e.g. casperJS, Behat, etc').
This looks like a great project! It looks pretty similar to PhantomCSS, but you're taking the pain out of hosting, which I appreciate: https://github.com/Huddle/PhantomCSS
One of my biggest struggles is introducing development practices like regression testing to traditional designers, a problem which is compounded by a lack of good tooling. Hopefuly Percy (and other tools like it) can help to alleviate this Problem.
I wonder if we could eventually even have TDD for designs. So once a component is designed in Photoshop, the first step would be to submit an image into Percy and then subsequent commits would check if the CSS had evolved to the point where it matched the original static design.
56 comments
[ 3.5 ms ] story [ 148 ms ] threadhttps://github.com/gabrielrotbart/gatling
I have some tweaks i was going to merge in that gave the diff's as %'s (so you could ignore small differences like fonts being different) as well as animate between the two versions. On my TODO was grabbing the actual element style attributes and then actually just diffing that, so the differences would be more or less spelled out.
Still, cool stuff!
But we really let ourselves down sometimes with regressions to the look and feel of the application.
Are there any competitor tools for this visual testing ?
It's written in Ruby but is not really dependent on Rails or anything to run. We use it to compare a list of URLs between the version we are currently testing in CircleCI and our production site. Any differences are highlighted in a nice gallery and we get a message in our Hipchat room.
[0]: https://github.com/xebia/VisualReview
I couldn't find a way to sign up and exclude private repos or organizations.
And yes it's a GitHub limitation, we request some limited readonly access and no code access (we don't need it).
How is that not a problem imposed by Percy?
I'd feel more comfortable having a different mechanism to request an invite.
While the permissions don't grant Code Access, it does ask to read Organization and Team membership. I'd expose data for every member of my companies github organization to Percy, JUST for an invite.
However:
Sign up for Service != Sign up for Invite.
Responsive capabilities will really improve usefulness, and the absence is somewhat of a deal-breaker. On responsive sites developers will have to eyeball each change even when using Percy, so checking the desktop resolution wouldn't be much additional effort relative to checking all other resolutions for each page.
On the biz-dev side, if it's going to be a paid product it might be a good idea to start charging ASAP!
It has been integrated with Travis CI and GitHub pull requests so that non-passing diffs are appended as a comment to the pull request by a bot account.
If there's an intended diff, a pull request can also automatically be submitted against the original pull request which, on merge, updates the "gold master" images and allows the original pull request to pass.
1.http://bbc-news.github.io/wraith/index.html
We're releasing our first version somewhere next week which includes a basic comparison GUI and testing api's. PR workflow planned in the release after that. Interesting to see we're not the only ones with this idea.
For example this uses Firefox, if all I change is something in the backend. My front end should look the exact same before the change and after.
Check out also http://shoov.io/ - there are a few blog posts, that explain the concepts and point to examples. Apart of the visual regression testing it also allows monitoring the live sites with your preferred functional testing frameworks (e.g. casperJS, Behat, etc').
Open source of course :)
One of my biggest struggles is introducing development practices like regression testing to traditional designers, a problem which is compounded by a lack of good tooling. Hopefuly Percy (and other tools like it) can help to alleviate this Problem.
I wonder if we could eventually even have TDD for designs. So once a component is designed in Photoshop, the first step would be to submit an image into Percy and then subsequent commits would check if the CSS had evolved to the point where it matched the original static design.