Quick example of how we use this at Segment... for each integration (https://segment.io/integrations) we need to create a vector logo. This got a bit tedious after the first 50 :) so we built nightmare + nightmare-swiftly so that our chat bot could "auto-generate" vector logos: https://cloudup.com/cEA-dTd3glM
Is it doing this by checking colored pixel areas and making SVG shapes for them? If so, whatever you use should replace Illustrator's Image Trace functionality, cause that is so bad at creating crisp vector copies of things.
Please note that because you used the base "phantom" library from npm it won't ever work with cluster (because node doesn't do the right thing with port=0 under cluster).
You'd have to switch to node-phantom-simple. Which might be worthwhile anyway because the phantom library you've used proved massively unstable at scale for us - I wrote it because of the need to work under cluster, and because dnode communication was bloated and unstable.
Yeah, there's a ton of "easier to use" libraries atop Phantom. CasperJS is by far the most recognized and community-backed. Not sure what value all of these others provide.
Ordinarily I would file a website bug, but this happens so often I'm mentioning it here to hopefully stop others from doing it, too.
Please don't use fancy quotes or the fancy apostrophe for code examples on your webpage. It looks out-of-place with the rest of the fixed width font and, most importantly, it stops folks from being able to copy-and-paste your examples into their terminal or editor.
I currently only have experience using CasperJS which also wraps-around PhantomJS. This has different goals (simpler) and therefore a simpler API, though. The use case at Segment (automatically generating vector logos from a given graphic) is even cooler though, I think! Thanks for sharing!
I do like the work they've done with Phantom but the API is horrendous and I found myself wrapping the parts I used. Nightmare looks more complete and probably better tested than my wrapper. Well done.
Shameless plug: I maintain Node-huxley (https://github.com/chenglou/node-huxley), which accomplishes what this project does, but without the need for writing automation code. Also compares previous/next screenshots for you automatically.
I am one of the founders of a startup that sells a similar, proprietary library for Python and Java: http://heliumhq.com. Maybe some of the people reading this will find it interesting.
Very nice. FYI best practices are to have a CTA or indicator on your site to scroll down, so no one leaves thinking the whole site is just an image and title.
Even after substituting the fancy ampersands on your homepage, and requiring nightmare your example still does not run:
var Nightmare = require('nightmare');
new Nightmare()
.goto('http://yahoo.com')
.type('input[title="Search"]', 'github nightmare')
.click('.searchsubmit')
.run();
yields:
node yahoo.js
phantom stdout: TypeError: 'null' is not an object (evaluating 'element.value = text')
phantom stdout: phantomjs://webpage.evaluate():3
phantomjs://webpage.evaluate():4
phantomjs://webpage.evaluate():4
stream.js:94
throw er; // Unhandled stream error in pipe.
^
TypeError: undefined is not a function
at next (/private/tmp/night/node_modules/nightmare/lib/index.js:56:23)
at f (/private/tmp/night/node_modules/nightmare/node_modules/once/once.js:16:25)
at Proto.apply (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/dnode/node_modules/dnode-protocol/index.js:123:13)
at Proto.handle (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/dnode/node_modules/dnode-protocol/index.js:99:19)
at D.dnode.handle (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/dnode/lib/dnode.js:140:21)
at D.dnode.write (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/dnode/lib/dnode.js:128:22)
at SockJSConnection.ondata (stream.js:51:26)
at SockJSConnection.EventEmitter.emit (events.js:107:17)
at Session.didMessage (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/shoe/node_modules/sockjs/lib/transport.js:220:25)
at WebSocketReceiver.didMessage (/private/tmp/night/node_modules/nightmare/node_modules/phantom/node_modules/shoe/node_modules/sockjs/lib/trans-websocket.js:102:40)
The examples in the githubs pages readme don't run either.
I am too lazy to debug this.
I've been wanting to programmatically get my transaction details from my bank for a while. I knew about phantomJS but never got around to trying it out. When I saw this I decided to give it a shot, and within half an hour I was able to log into my bank's website (a complicated multi-step process that I had failed to achieve with just POST requests).
Nice tool, thanks. The debugging could use a bit of work - whenever something went wrong it just spit out a bunch of "null is not an object" messages from phantom. Overall it was very easy to use though.
Whats a good way to automate this to run? I see the getting started in the documentation says "Save it as hello.js and then run it from the command line" but Iwasnt sure if there are tools like CRON in Linux to automate the script.
That is the documentation for phantomjs, which is a stand-alone process. Nightmare is a package for nodejs (that interacts with a phantomjs process), so your code runs in a node context, where there are plenty of scheduling options, e.g. the node-cron package, or just plain setInterval().
I really want to do this with the two accounts I have at different banks. Neither makes it easy to automatically download my transactions. What I want is to run a command to run to download the latest transactions and output some simple ledger-like[1] output.
When I have tried to do double entry account before it was tricky to keep up. I would spent a lot of time trying to figure out what I had already reconciled and what I had not.
the problem with Phantom.js is that compared to an actual browser it still falls short. I've given up on Phantom.js all together because it is actually not a true replacement for Selenium running on an actual browser. I found it far more reliable to have Firefox running on Xvfb for when I'm testing without worrying about Phantom.js behaving wildly.
You can use Selenium for automation and web page interaction.
There is just an overhead of running Xvfb but it's trivial to setup.
What I mean is that PhantomJS engine still falls short of a real browser. Advantage of running a real browser with Xvfb is that you won't ever have to worry about if the page is going to render properly or have it behave the way you expect it to. The overhead is not of a big deal, in fact it's a tradeoff for rendering accuracy.
Don't get me wrong headless browser like PhantomJS is headed in the right direction but if compatibility with website is crucial for your success, it is not enough and I just don't have a lot of confidence that it will replace a real browser with Xvfb.
Basically the Javascript libraries are reinventing a lot of the wheels that have already been done in other languages before the explosion of Node.js.
firefoxdriver already exists in many other languages, I don't know what is the point of writing it all over again for javascript. The callback stuff gets messy. I still much prefer Python or even Java to write my tests.
I can absolutely concur that it crashes frequently. I have high hopes v2 fixes this. Most of the crashes are due to WebKit bugs that have since been resolved in QT5.
I've reached the same conclusion for our suite of UI tests (which are based on Cucumber + Capybara + SitePrism) - Phantom has random issues that don't occur with any other browser/driver (e.g. elements not being found even if you add a long wait - our app is rich client side Angular so I suspect some JS issue is happening) and it's too hard to debug what is going on compared to Firefox + Xvfb (where you can just redirect the display to a real X display and see what is going on).
Yes, with selenium you can actually watch the entire test happen and see where things go wrong. Very easy to debug and understand but PhantomJS makes it really tough to find out why something failed because its headless. Sometimes I would try to take screenshots to figure out what was going on and even that would fail at times.
Maybe you'd be able to run more instances with PhantomJS compared to running Xvfb and a real browser because of lower memory consumption. I don't know if this is true.
As someone entirely unfamiliar with both Nightmare and PhantomJS, what is the typical use case? It isn't obvious to me from the examples or FAQs. Why prefer these over something like Selenium?
76 comments
[ 1005 ms ] story [ 984 ms ] threadHope it's helpful!
You'd have to switch to node-phantom-simple. Which might be worthwhile anyway because the phantom library you've used proved massively unstable at scale for us - I wrote it because of the need to work under cluster, and because dnode communication was bloated and unstable.
Swiftly lead dev here. Just wanted to say this is very cool! I was a little bit surprised to see us in your examples.
We've been looking into improving our end-to-end testing tools, but it looks like you guys have already taken care of it. Nice work! ;-)
With Selenium you have full cross browser support, while Phantom is Webkit based.
Please don't use fancy quotes or the fancy apostrophe for code examples on your webpage. It looks out-of-place with the rest of the fixed width font and, most importantly, it stops folks from being able to copy-and-paste your examples into their terminal or editor.
If so, that seems super easy and more straightforward than vanilla Shadow or even Casper.
By default there is no indication of completion, hence the log.
https://github.com/ianmiell/shutit/blob/master/library/night...
which should help matters. Will reply when done.
Nice tool, thanks. The debugging could use a bit of work - whenever something went wrong it just spit out a bunch of "null is not an object" messages from phantom. Overall it was very easy to use though.
When I have tried to do double entry account before it was tricky to keep up. I would spent a lot of time trying to figure out what I had already reconciled and what I had not.
1: http://www.ledger-cli.org/
What I mean is that PhantomJS engine still falls short of a real browser. Advantage of running a real browser with Xvfb is that you won't ever have to worry about if the page is going to render properly or have it behave the way you expect it to. The overhead is not of a big deal, in fact it's a tradeoff for rendering accuracy.
Don't get me wrong headless browser like PhantomJS is headed in the right direction but if compatibility with website is crucial for your success, it is not enough and I just don't have a lot of confidence that it will replace a real browser with Xvfb.
Basically the Javascript libraries are reinventing a lot of the wheels that have already been done in other languages before the explosion of Node.js.
Maybe you'd be able to run more instances with PhantomJS compared to running Xvfb and a real browser because of lower memory consumption. I don't know if this is true.
Browser automation? Is this a test suite of some sort, something like selenium? Or is it more like Mechanize?