HN Explorer

Ask HN: What should I know regarding Scraping Websites?

2 points by deadcoder0904 ↗ HN
Sometimes crowd-sourcing takes time & if you already have a market you can get a lot of data for free via scraping.

So what should I know regarding Web Scraping?

Not all websites provide API, so many people scrape content.

What are the legal implications?

Have you scraped any? If so, anything you've learned the hard way?

Also, is there anyway to prevent website from scraping content?

3 comments

[ 2.8 ms ] story [ 17.4 ms ] thread
Petrakis
Check the robots.txt and the ToS of the site and should be fine. In case of doubt, you could contact the site administrator.
lifencoder
"Also, is there anyway to prevent website from scraping content?"

You can implement a cloudflare dns so that people(or bots) are not able to scrape your content. even you can have robot authentication checking methods like google recaptcha to prevent scraping of content.

krapp
>You can implement a cloudflare dns so that people(or bots) are not able to scrape your content. even you can have robot authentication checking methods like google recaptcha to prevent scraping of content.

There are entire captcha farm industries dedicated to 'automating' the filling out of captchas using mTurk and similar crowdsourcing services for nearly nothing,to say nothing of exploits for the captchas themselves.

Nothing can be done to prevent the scraping of web content, at all. If a site can be loaded into a browser, it can be scraped. It's just a matter of the result being worth the effort.