4 comments

[ 3.3 ms ] story [ 16.3 ms ] thread
Interesting, but the title is confusing. Better to use the original in this case.
Is there a generic list of defense techniques against scrapers, including but not limited to Google? E.g. move some content behind a login wall?
In theory, you can disallow crawlers and bots via robots.txt. It's not a hard defense per se because crawlers can ignore it.