3 comments

[ 3.1 ms ] story [ 21.2 ms ] thread
honeypot entry to ensure that crawlers are complying with robots.txt?
I don't get it. The link points to www.amazon.com/robots.txt.
if you look in the file you'll see a Disallow entry for /product-reviews/B0069IY63Y.