3 comments

[ 4.5 ms ] story [ 12.8 ms ] thread
honeypot entry to ensure that crawlers are complying with robots.txt?
I don't get it. The link points to www.amazon.com/robots.txt.
if you look in the file you'll see a Disallow entry for /product-reviews/B0069IY63Y.