Nike.com's robots.txt file (nike.com)

Odd to prohibit crawling PDFs; modern search engines handle those just fine.

just crawl it...expect SWFs and PDFs.

In Firefox I got access denied... In case anyone else can't get to it, here's a pastebin: http://pastebin.com/raw/amTGwEEa

Not even disallowing crappy crawlers like majestic 12? They're going to scuff your Jordans!

thats pretty great

That's a nice work.