Ask HN: How to effectively block scrapers?

1 points by jorgecurio ↗ HN
Let's leave the ethics debate aside and ask this question: is it realistic to invest time and money into blocking scrapers?

I see distill networks is popular, but what is their pricing? try to scrape crunchbase and see what happens. It's actually very good I tested with phantomjs, selenium + firefox and it's able to detect and successfully block the traffic.

I'm willing to invest time and money into preventing competitors from launching based on scraping my data that I worked hard for.

Is this something that I can implement? Sounds complicated in two parts

1) building fingerprints on user agent and other information. What do you include, where do you start?

2) Distinguishing malicious requests on a time series while avoiding false positives (some people compulsively click next button twice if it doesn't load immediately

What are some surefire ways that will give me high degree of confidence that scraper will fail.

0 comments

[ 3.2 ms ] story [ 11.9 ms ] thread

No comments yet.