[–] ithkuil 15y ago ↗ Well, you can try it with:wget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)and you get the same error.I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result
[–] andrenotgiant 15y ago ↗ they have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers to go away.Maybe it's their way of playing nice with all of the merchants they are listing.
5 comments
[ 2.8 ms ] story [ 12.3 ms ] threadwget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
and you get the same error.
I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result
Maybe it's their way of playing nice with all of the merchants they are listing.