For all the many problems with X11, it's still pretty cool that you can run something as complicated as Chrome in a headless manner just by running another program (xvfb) that speaks the protocol.
(I keep intending to hack Blink into a state where it can run truly headless, like a library, so that hacks like the above aren't necessary.)
I've experimented with running headless tests using Firefox, in hopes of getting a good performance boost in terms of test speed. The results actually did not indicate much of a difference. If anything, FF in headless mode on CentOS ran slower than on Windows.
That's not to say there aren't good use cases for running headless browsers with Selenium. One huge benefit is not needing to run Windows VMs.
Nice. I would add that Selenium performance isn't super awesome but if you only need to use it for a specific set of integration tests it's OK. You can use something like PhantomJS to speed through the rest your tests that don't specifically need exact browser implementations.
How does it differ from PhantomJS/CasperJS (headless webkit)? I have been using these for a while and they can do pretty much everything that you can do in a browser.
I have a 1/2 written post about this exactly. I took it out because it would have made this post too long. But TLDR: I found a few edge cases where using a real browser was easier to use than a headless toolkit. Definitely will be my next post.
The point of testing in a real browser is to test in a real browser because your users use a real browser. Testing in a headless browser is faster, but proves nothing.
One odd issue that I ran into was that Chrome (and Firefox) wouldn't always go fullscreen. I had to install a window manager that defaulted to fullscreen. In particular, I went with Ratpoision: http://danielwebb.us/software/ratpoison/
One thing I always think is worth mentioning in posts like this is that the above setup (and any setup using Selenium, really) is going to be super slow. More than one order of magnitude slow, when compared to direct HTTP requests.
It's only ever worth using Selenium when you literally can't do anything else to get your testing done.
Because it has to load a virtual frame buffer (Xvfb), the chrome browser, a Selenium server, and then your testing framework. All that adds up to crap performance. There's a lot of pieces that could be simplified with a smaller set of requirements. But like @diminoten said: it's worth it if you have no other choice.
I've had good luck with a similar setup (somewhat inspired by IMVU's) doing integration testing on an Ember app: Chromium + chromedriver inside a VirtualBox Vagrant instance using Xvfb + ratpoison + x11vnc. My tests run from a modified version of Wai Yip Tung's HTMLTestRunner (Python unittest module underneath) so reloading the tests tab in a browser on my dev machine reruns the tests. All of the tests are written in Python. If something gets stuck you open up a VNC client and poke around manually. It's a bit of work to set up and the tests can be a bit slow but I'd recommend it when you really need to test in a browser.
15 comments
[ 3.1 ms ] story [ 44.5 ms ] thread(I keep intending to hack Blink into a state where it can run truly headless, like a library, so that hacks like the above aren't necessary.)
That's not to say there aren't good use cases for running headless browsers with Selenium. One huge benefit is not needing to run Windows VMs.
PhantomJS also runs some ancient version of WebKit, which means it is not compatible with modern browsers.
It's only ever worth using Selenium when you literally can't do anything else to get your testing done.
Edit: Going through my bookmarks bar and found this, from 2008 - http://www.alittlemadness.com/2008/03/05/running-selenium-he...
Might help anyone if they run into issues.