How about networking improvements? I gave up on PhantomJS a long time ago because it did not support SNI and was therefore unable to talk to most HTTPS sites. Is that situation better now with this new release and the newer Qt underneath?
The trouble with Nightmare and Electron is that they can't be run truly headless, you need to use xvfb with its additional overhead.
There are various requests to have electron made truly headless but its not a core feature that most users want. However we may see it supported (I hope) when the new headless features of Google Chrome make there way down into libchromiumcontent then electron.
This is just my opinion, but I can see PhantomJS lose a lot of its appeal when the --headless switch for Google Chrome hits the stable channel (quite a way off still, sure). This was already discussed to some extent here: https://news.ycombinator.com/item?id=11839303. Add a couple of porcelain libraries on top of chrome-remote-interface, and you have an even more powerful instrumented headless browser, plus you don't need to worry about security, performance and keeping up with crazy new web features. Also, Chrome headless will encompass features aimed at automation/CI like:
- "Virtual" time (fast-forward time)
- Session isolation (even for same origin)
- Full page screenshots out of the box
Shameless plug: I'm using Google Chrome to drive https://urlscan.io and will make the switch to the headless version once it's included in the Beta channel.
(Disclosure: I'm a developer advocate for chrome so I'm heavily biased, but I love everything people are doing g with browsers on the server https://paul.kinlan.me/the-headless-web/ )
I always disliked working with Phantom. Slow, buggy, crashy, limiting and weird subset of Javascript support with an awful ecosystem of hacks trying to get it usable in Node. I can't wait until a more stable tool superceeds it.
Casperjs, which runs on top of PhantomJS has been nicer to use for me. Also Nightmare(http://www.nightmarejs.org/). The problem I always had with PhantomJS is that it's more difficult than it should be to automatize very simple website interactions.
In Nightmare's homepage you can see a comparison between a script made in PhantomJS and one in Nightmare.
i built chromium headless_shell for ubuntu 16 in the last 3 months to compare it to my phantomjs 2.1.1 build (cookie patch and ghostdriver selenium 2.53 patch). i was disappointed in what chromium headless_shell had to offer.
good: slightly faster on javascript heavy pages, full dowload and upload support, selenium support for alerts/popups
bad: slower startup, slower average performance, no mobile support, no screenshot support, takes at least 2x cpu and memory compared to phantomjs
for me phantomjs/ghostdriver is faster than chromedriver. plus chromedriver has an annoying click "feature"[1] that can break selenium tests.
I'm working on a wrapper to completely expose the chrome remote debug interface in python [here](https://github.com/fake-name/ChromeController), if you're interested. In doing so, I've found (and reported) several bugs in the remote interface documentation as well. It's a fun project.
I wish the PhantomJS project was more upfront about the version of Qt (and therefore Webkit) that ships with each version, like they are in this email. That information is very hard to track down otherwise, and I think it's the most crucial part of each release (since feature support in Phantom tends to lag browsers by several years).
Additionally, it would be cool for sites like caniuse or MDN to track Qt Webkit feature support as well.
Congrats and thanks to the phantomjs devs for dedicating their time to develop this useful software.
I've used PhantomJS for web app testing, but not as general-purpose browser automation tool. Imperfect, insufficient, and under-maintained as it may be, I appreciate the existence of the Webdriver API and its main implementation based on PhantomJS. It defines a baseline model for web test case development, and a choice of implementations for test execution (recently also directly supported by Safari, I believe).
I haven't used the new version of QtWebKit yet, but I've heard the changes are pretty huge and they basically bring QtWebKit back up to par with other modern web browsers. It's pretty exciting to see one of the lesser known web browser engines get development like this, especially since it results in improvements to things like qutebrowser (and PhantomJS.)
I'm hoping this will eventually speed up my tests during deployment a bit as I currently have to pipe my ES2015 code through a Karma transform to have it readable by my PhantomJS instance.
I have really benefited from using this tool on a couple of side projects. If any Devs are reading, thank you for continuing to work on such a useful tool!
Working on this - Dullahan which is a headless browser SDK based on Chromium Embedded Framework and hope to have something like PhantomJS working soon albeit using a modern browser rendering engine.
The latest 2.5.0 beta doesnt have the updated ghostdriver yet.
How much ram does chrome --headless use (true headless not XvfB), because I want to run about 200 of these in parallel and keeping the ram down is important obviously.
Does anyone have any suggestions? I figure by using selenium grid I can run them in docker containers and even on multiple hardware machines.
But its hitting some kind of system limits or sync issues because each process is started before run time rather than ready and waiting in a pool.
Any help on running 200 headless browsers via selenium grid would be greatly appreciated. I think the current selenium chrome Xvbf docker container uses like 300mb+ per instance so that would be 60gb of ram which is a fair bit.
The latest 2.5.0 beta doesnt have the updated ghostdriver yet.
How much ram does chrome --headless use (true headless not XvfB), because I want to run about 200 of these in parallel and keeping the ram down is important obviously.
Does anyone have any suggestions? I figure by using selenium grid I can run them in docker containers and even on multiple hardware machines.
But its hitting some kind of system limits or sync issues because each process is started before run time rather than ready and waiting in a pool.
Any help on running 200 headless browsers via selenium grid would be greatly appreciated. I think the current selenium chrome Xvbf docker container uses like 300mb+ per instance so that would be 60gb of ram which is a fair bit.
38 comments
[ 2.9 ms ] story [ 88.0 ms ] threadWe're currently using PhantomJS for some headless stuff, but are considering a switch.
There are various requests to have electron made truly headless but its not a core feature that most users want. However we may see it supported (I hope) when the new headless features of Google Chrome make there way down into libchromiumcontent then electron.
- "Virtual" time (fast-forward time) - Session isolation (even for same origin) - Full page screenshots out of the box
Slides for Chrome headless: https://docs.google.com/presentation/d/1gqK9F4lGAY3TZudAtdcx...
Shameless plug: I'm using Google Chrome to drive https://urlscan.io and will make the switch to the headless version once it's included in the Beta channel.
(Disclosure: I'm a developer advocate for chrome so I'm heavily biased, but I love everything people are doing g with browsers on the server https://paul.kinlan.me/the-headless-web/ )
I wrote something similar to serve low-bandwidth (pre-readabilitied pages) to my parents low-bandwidth internet over a MITM proxy.
I'm automating in environments that don't have X.
good: slightly faster on javascript heavy pages, full dowload and upload support, selenium support for alerts/popups
bad: slower startup, slower average performance, no mobile support, no screenshot support, takes at least 2x cpu and memory compared to phantomjs
for me phantomjs/ghostdriver is faster than chromedriver. plus chromedriver has an annoying click "feature"[1] that can break selenium tests.
[1] https://sites.google.com/a/chromium.org/chromedriver/help/cl...
I'm working on a wrapper to completely expose the chrome remote debug interface in python [here](https://github.com/fake-name/ChromeController), if you're interested. In doing so, I've found (and reported) several bugs in the remote interface documentation as well. It's a fun project.
I hope to eventually completely replace phantomjs in my automated scraping project with headless chromium, but there's still some [strange issues](https://bugs.chromium.org/p/chromium/issues/detail?id=672744).
PhantomJS works well enough, 2.5 will hopefully work better. `--headless` will be even better than that, but it isn't here yet.
Additionally, it would be cool for sites like caniuse or MDN to track Qt Webkit feature support as well.
https://rm891swy536139ttwzee.mnutt.sandcats.io/853e0e85-170e...
I specifically left out some audio/video support from my build because I didn't need it. And Qt 5.8 will be released this month with HTTP/2 support.
I've used PhantomJS for web app testing, but not as general-purpose browser automation tool. Imperfect, insufficient, and under-maintained as it may be, I appreciate the existence of the Webdriver API and its main implementation based on PhantomJS. It defines a baseline model for web test case development, and a choice of implementations for test execution (recently also directly supported by Safari, I believe).
For an exhaustive list of alternatives, also see: https://github.com/dhamaniasad/HeadlessBrowsers
I'm hoping this will eventually speed up my tests during deployment a bit as I currently have to pipe my ES2015 code through a Karma transform to have it readable by my PhantomJS instance.
Working on this - Dullahan which is a headless browser SDK based on Chromium Embedded Framework and hope to have something like PhantomJS working soon albeit using a modern browser rendering engine.
I tried this one for example but it just crashes randomly when the jobs are running: https://github.com/AkeemMcLennon/docker-selenium-node-phanto...
The latest 2.5.0 beta doesnt have the updated ghostdriver yet.
How much ram does chrome --headless use (true headless not XvfB), because I want to run about 200 of these in parallel and keeping the ram down is important obviously.
Does anyone have any suggestions? I figure by using selenium grid I can run them in docker containers and even on multiple hardware machines.
Currently, I am running about 50 in parallel with Magellan https://github.com/TestArmada/magellan
But its hitting some kind of system limits or sync issues because each process is started before run time rather than ready and waiting in a pool.
Any help on running 200 headless browsers via selenium grid would be greatly appreciated. I think the current selenium chrome Xvbf docker container uses like 300mb+ per instance so that would be 60gb of ram which is a fair bit.
I forked the binary from https://mecrazy.net/2016/06/06/i-built-phantomjs-2-1-1-for-r...
it's available here: https://github.com/ShamariFeaster/phantomjs-binaries
I tried this one for example but it just crashes randomly when the jobs are running: https://github.com/AkeemMcLennon/docker-selenium-node-phanto...
The latest 2.5.0 beta doesnt have the updated ghostdriver yet.
How much ram does chrome --headless use (true headless not XvfB), because I want to run about 200 of these in parallel and keeping the ram down is important obviously.
Does anyone have any suggestions? I figure by using selenium grid I can run them in docker containers and even on multiple hardware machines.
Currently, I am running about 50 in parallel with Magellan https://github.com/TestArmada/magellan
But its hitting some kind of system limits or sync issues because each process is started before run time rather than ready and waiting in a pool.
Any help on running 200 headless browsers via selenium grid would be greatly appreciated. I think the current selenium chrome Xvbf docker container uses like 300mb+ per instance so that would be 60gb of ram which is a fair bit.