HN Explorer

Ask HN: Does robots.txt serve a real purpose?

4 points by mgalka ↗ HN
Robots.txt is treated like a fundamental piece of SEO, but I have never seen a straight answer on what its purpose is.

There are clearly lots of ways it can hurt SEO if used incorrectly, but is there a scenario where a site gets more organic traffic because of robots.txt than it would without it?

Often, people say it should be used to hide duplicate content. However, Google has been pretty clear that duplicate content is only penalized when it is intended to deceive. And they explicitly say not to use robots.txt to block duplicate content.

>Google does not recommend blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods.

https://support.google.com/webmasters/answer/66359?hl=en

It is also a poor way of blocking sensitive pages that you don't want to appear in the SERPs. A noindex meta tag is better.

Is there a good example where robots.txt helps SEO, or where it serves any other real purpose?

3 comments

[ 0.27 ms ] story [ 17.4 ms ] thread
geekishmatt
robots.txt is the "(Web) Robots Exclusion List".

Any web robots should parse your /robots.txt before accessing any site, but it's still the robot's (or its programmer's) choice to use it.

So: No, but still add it to your site and maintain it.

tonyle
My opinion is to view it as an optional thing to help other crawlers( and bored humans) index your content. You can't really use it to block crawlers since bad crawlers will just ignore it.

For example, if I wanted to download every single one of Microsoft knowledge base article, I could go to https://support.microsoft.com/robots.txt and download every url in the sitemap.

Compare that with other sites that don't provide a sitemap so I would have to hit every single page and try to find out all the links manually by downloading, parsing for links, etc.

Less work for me and less load for the site.

While I"m not sure if it has a direct impact on seo, Making your site more accessible is only a good thing imho.

mgalka
Sounds like you are speaking specifically about the sitemap. Agree that they serve a good purpose, though you don't need robots.txt to have a sitemap.