The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.
Having worked on bot detection in the past. Some really simple old fashioned attacks happened by doing the opposite of what the robots.txt file says.
While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
That was the first thing that I have learnt about the robots.txt file. Even RFC 9309 Robots Exclusion Protocol document: https://www.rfc-editor.org/rfc/rfc9309.html - mentions:
> These rules are not a form of access authorization.
Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really.
I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.
I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google. It isn’t just about that preview for LinkedIn, it’s about making sure the web is accessible by everyone and everything that is trying to make its way. Sure, block the malicious ones. But don’t just assume that every bot is malicious by default.
The most annoying thing about being a good bot owner, in my experience, is when you get complaints about it misbehaving, only to find that it was actually somebody malicious who wrote their own abusive bot, but is using your bot's user agent.
I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes with an intentionality behind it. If you want to allow or disallow certain traffic, you also should answer the question why or why not. That requires understanding what each bot does. That takes time and effort.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
> That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen.
The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
I still see the value in robots.txt and DNT as a clear, standardised way of posting a "don't do this" sign that companies could be forced to respect through legal means.
The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR.
The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.
LinkedIn is by far the worst offender in post previews. The doctype tag must be all lowercase. The HTML document must be well-formed (the meta tags must be in an explicit <head> block, for example). You must have OG meta tags for url, title, type, and image. The url meta tag gets visited, even if it's the same address the inspector is already looking at.
Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
What astounds me is there are no readily available libraries crawler authors can reach for to parse robots.txt and meta robots tags, to decide what is allowed, and to work through the arcane and poorly documented priorities between the two robots lists, including what to do when they disagree, which they often do.
Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not.
Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
The "good" bot writers rarely have enough resources to demolish servers blindly, and are generally more careful whether or not you make it easier, so there's not much incentive.
I try to stay away from negative takes here, so I’ll keep this as constructive as I can:
It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight.
I agree, and I am also confused on how this got on the frontpage of all things.
It's like reading a news article of 'water is wet'.
You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works.
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
if you are hosting a house party that invites the entire world robots.txt is a neon sign to guide guests to where the beers are, who's cooking what kind of burgers and on what grill; rules of the house etc - you'll still have to secure your gold chains and laptop in a safe somewhere or decide to even keep them in the same house yourself
Gold chains etc should be behind authentication. Robots txt is more like a warning sign that says the hedgemaze in the garden goes on forever, so probably stay out of it.
This doesn't seem like a new discovery at all - this is what news publications have been dealing with ever since they went online.
You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
I believe that as search engines continue to move toward AI summaries and responses, it will reduce the traffic to the websites since most people will be ok accepting the answers that the AI gave them.
My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.
I think a paragraph could have been enough to describe the issue.
My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms.
2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me.
I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search.
I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.
This reminds me of an old friend of mine who wrote long revelation posts on how he started using the "private" keyword in C++ after compiler helped him to find why a class member changed unexpectedly and how he no longer drives car with the clutch half-pressed because it burns the clutch.
The problem with robots.txt is the reliance on identity rather purpose of the bots.
The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies.
What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival".
And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
The "full solution" to this, of course, is micropayments. A bot which has to pay a tenth of a cent to you every time it visits one of your pages or something else the page 404s will quickly rack up a $10 bill crawling a whole 10,000 page site. If it tries to do that every day, or every hour, that's an excellent payday for you and a very compelling reason for almost all bots to blacklist your domain name.
A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that.
The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
Weirdly, this is something that Apple actually gets right - the little „previews” you get when sharing links in iMessage get generated client-side; _by the sender_.
There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
This is a kind of scream test, even if self-inflicted. Scream tests are usually a good way to discover actual usage in complex (or not so complex) systems.
Another common unintended consequence I've seen is conflating crawling and indexing with regards to robots.txt.
If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index.
If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled!
Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.
44 comments
[ 13.4 ms ] story [ 81.7 ms ] threadWhile I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
> These rules are not a form of access authorization.
Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really.
I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen.
The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR.
That was a theory at least...
https://en.wikipedia.org/wiki/Robots.txt
The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.
Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to know though.
Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not.
Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight.
And then they posted it here for attention.
You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.
My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.
1)
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews.
2)
From your first post,
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms.
2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me.
I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search.
I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.
If you don't want people to crawl your content, don't put it online.
There are so many consequences of disallowing robots -- what about the Internet Archive for example?
The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies.
What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival".
And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
I wish there were way less posts like this.
A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that.
The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
It is amazing what people think is important these days.
If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index.
If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled!
Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.