from the github page:
Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML.
If you really want to scrape pages, you should use something like https://github.com/chriso/node.io/ which batches things in jobs, helps with error handling, io, etc...
Does Node have anything like Mechanize? Handling cookie state and such is something that is much more useful than the selector functionality of jQuery...which is great, but not any better than what Nokogiri offers.
http://zombie.labnotes.org/ is a library I've used with great success. The documentation in particular is cute.
I found PhantomJS unnecessarily convoluted for trivial tasks and was unable to figure how to do the nontrivial thing I was actually trying to do. The documentation in particular was unusable.
Using JQuery server-side to process Twitter posts which are already in JSON format is just so dumb I can believe I read an entirely usless blog post about it.
I have also found node+jQuery an effective web crawling combination. In particular the cheerio library https://github.com/MatthewMueller/cheerio greatly simplifies data extraction. And as others have mentioned, the asynchronous nature of node is perfectly suited to crawling (as long as you take care not to accidentally DDOS the target site).
14 comments
[ 1.7 ms ] story [ 41.0 ms ] threadNot that I'm advocating this, but a potential advantage of scraping, is it can be combined with tor or proxies, etc to get around limits.
'''
* Node.js [...]
* jQuery [...]
[...]
This approach has become my hammer when web scraping tasks come up.
'''
If all you have is a hammer, you may find yourself noticing that objects become more nail-like :).
I found PhantomJS unnecessarily convoluted for trivial tasks and was unable to figure how to do the nontrivial thing I was actually trying to do. The documentation in particular was unusable.