Ask HN: Web Scraping with Node.js

Anyone out there have a cachet of good Node web scraping libs, and/or other resources? Opinions?

I have the intuition that my Google searches are a bit inadequate. Thank you!

6 comments

[ 2.7 ms ] story [ 22.0 ms ] thread

Should it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.

jneumann004 10y ago ↗

I haven't used this for web scraping, but have you tried out http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.

Nilef 10y ago ↗

Try this out https://github.com/IonicaBizau/scrape-it

iends 10y ago ↗

Check out https://github.com/cheeriojs/cheerio

tmpfs 10y ago ↗

I found https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown - a cli for validating web pages etc.

paulzerkel 10y ago ↗

Here is a place to start: https://github.com/lorien/awesome-web-scraping/blob/master/j...

You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.