Ask HN: Examples of interesting robots.txt

1 points by r4vik 14y ago ↗ HN

I'm sitting here going through some of our competitors robots.txt files wondering if they were doing anything interesting or innovative but they all seem pretty run of the mill. HN can you share some robots.txts that do something unusual or innovative?

2 comments

[ 5.0 ms ] story [ 12.1 ms ] thread

balducci 14y ago ↗

How "innovative" do you expect robots.txt to be?

r4vik 14y ago ↗

Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.