Ask HN: How to effectively block scrapers?
I see distill networks is popular, but what is their pricing? try to scrape crunchbase and see what happens. It's actually very good I tested with phantomjs, selenium + firefox and it's able to detect and successfully block the traffic.
I'm willing to invest time and money into preventing competitors from launching based on scraping my data that I worked hard for.
Is this something that I can implement? Sounds complicated in two parts
1) building fingerprints on user agent and other information. What do you include, where do you start?
2) Distinguishing malicious requests on a time series while avoiding false positives (some people compulsively click next button twice if it doesn't load immediately
What are some surefire ways that will give me high degree of confidence that scraper will fail.
0 comments
[ 3.2 ms ] story [ 11.9 ms ] threadNo comments yet.