Ask HN: mad search bots?

3 points by xentronium 15y ago ↗ HN

I've got a groupblog-like site (somewhat niche, I must notice) which got articles and tags. Watching the stats I found out that about 80% of time and like 250+ reqs/minute application serves are tag pages for searchbots (from different search engines).

Am I doing something wrong?

I don't mind search bots since they hit the cache, but they seem to crawl the same content over and over again. Isn't that weird?

1 comment

[ 0.29 ms ] story [ 11.1 ms ] thread

JoachimSchipper 15y ago ↗

Microsoft's bot used to do that kind of thing - not sure if it's been fixed yet. AFAIK, it's just stupid and there's no way to fix it.

You can set crawl-delay in robots.txt if you want to reduce load, but you don't appear to need it. Note that, again, MS' bot doesn't honour this.

It may be sensible to make the tag pages noindex, and a sitemap.xml may help these bots. But I have no other ideas.