Presumably you'd want Google to index pages, just not try to discover pages by brute forcing the URL?
Is there any way to do that in robots.txt? Someone did mention adjusting the crawl rate in Webmaster Tools -- would the author need to do this for every search engine?
Submitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the blog post. Lots of sitemaps are incomplete or outdated or completely invalid or whatever.
I feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.
robots.txt can certainly keep google away from your search box, but it won't defend you against the lower-case probes, the extra letter probes, the delete-a-character probes, etc.
Yet he didn't try it. That's seriously weird that he doesn't mentions trying that (he updated his post to say they doesn't support regex, yet does support wildcard, which is all what's needed for a search).
He removed the search before using a simple robots.txt...
The first wave of the attack began dropping off letters.
I wouldn't call it an attack, but it seems more like some sort of autocompletion checking given how similar it is to the queries generated to Google if you use their search autocomplete feature --- one request for each character entered.
That said, Google's search quality has already taken a steep decline and I've been getting blocked very often for searching more obscure things now, so whatever "bot mitigations" they put in, I hope they don't make that even worse.
The claim was that it started with the full URL and progressively removed them. If it were reversed I would buy the autocomplete bug. I fixed something like that a few weeks ago at work.
If Chrome address bar autocomplete generates queries (which I'm not sure if it does), then the following scenario could easily explain what's going on:
- As a user, I type "foo.com" into my address bar. Having already gone to a specific page of that site before, autocomplete "helpfully" expands it to that page's URL "https://foo.com/this/is/a/page"
- It turns out I don't actually want that, I want the main page, so I start deleting characters until I'm left with "https://foo.com"
If Google took each of those deletions as a separate URL (either by bug or on purpose) and tried to verify said URL is valid, I could see a GoogleBot request being generated for each iteration between the initial URL and the full URL, starting with that initial one.
Given what I've read on the article, I'm fairly confident that what's happening here is a conjunction of Chrome autocomplete suggestions causing the crawl of every URL permutation and the website returning the same content to every permutation, instead of using 403 redirects or including a `rel=canonical tag`.
I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails.
Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something.
I can't help but wonder if this might be a similar case.
The imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches.
The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).
> The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).
If you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages.
Curiously, I've seen similar posts in various hacker blogs a couple of times. It's typically a very stereotypical hacker, running his own website from a machine that he personally administers, that runs only free software. Because his (and, unsurprisingly, all of them are men) other hobbies are very specific and rare, and he's very competent in them, he getting a reasonable amount of search traffic, but never a lot of comments.
I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.
On one of my obscure but public domains I always have fun seeing Chinese traffic guessing the admin password of the VPS at least 3x a day. I guess somebody is continuously poking at vulnerabilities to get root access to as many domains as they can, then later maybe selling them to their "customers"? One doesn't have to be paranoid these days...
True, you'll get all kinds of weird traffic like that running public services. Especially hidden services on Tor where people generally aren't backed by CDNs and attacks can be made anonymously.
But it's weirder to claim that Googlebot is aggressively DDoSing you.
Well, there's always the saying "It's not paranoia if they're really out to get you." And DOS attacks of many flavors are employed early and often against any public site.
And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.
He's complaining that Google is sending 100 requests per hour to his site (i.e. one request every 36 seconds).
That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.
The reason it's generating 100 alerts per hour is that his server alerts on every request to a "bad" (non-existent?) URL:
> My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack.
And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.
Googlebot does sometimes submit search forms in an attempt to discover new pages, though I'm surprised to hear "hundreds of thousands". Either that's an exaggeration or it was over a very long time period.
As other comments mentioned, he could have easily prevented this with robots.txt:
Would that work if the search box was on the main page, though? Presumably, GoogleBot wouldn't index the results, but is it smart enough to know that the form redirects to the disallowed page?
He doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into the ground.
The main issues and complaints we run into are mostly unusual requests for indexing content.
People see the total number of requests and think it's a significant burden and just have an irrational need to protect their content.
It's almost a form of paranoia at some point.
I mean we have NO problem removing someone that doesn't want to be indexed and 99/100 we do so without a problem.
The only times we push back are when government organizations contact us as we feel we have a legal right to index the content. We haven't had to really fight over this because they usually agree and that's the end of the discussion.
I see hundreds of thousands of requests from search engines daily without any burden, but I do block all other bots. It has nothing to do with paranoia, a lot of bots are just evil making it impossible to act towards bots in good faith. For example there are forensic bots, copyright protection bots, brand protections bots and similar bots with the whole purpose of using crawled data to send you threats and "abuse" reports or drag you to court. There are bots that scan for vulnerabilities to exploit and bots that scan pages to find something bad to blacklist somewhere. There are bots that generate spammy links or even grab content for black hat seo, something search engines punish you for. There are bots that belong to companies that sell your data or try to gain competitive advantage through this without giving you anything in return, something you may or may not be comfortable with. There are even bots that gather data for political and intelligence purposes, literally causing death and destruction.
I had no idea that googlebot (and others) will sometimes POST to urls[0] - this seems like a terrible idea.
I imagine what happened to this guy is that somebody coded up a site that used his as a service and google spidered it and went crazy trying to follow the links. It sucks, but that is what robots.txt is for. Anything that requires more than just serving a page should be blocked.
You can drive yourself insane reading your logs trying to guess what bots are trying to achieve. They seem to excel at spidering exactly what you don't want while ignoring the content that would actually be useful.
The one time I did have a problem with a misbehaving crawler was with something called 80legs[1]
I'm sure their ML is pretty decent... but I do have to wonder if it's ever wound up leaving hundreds of comments on an anonymous forum or something, thinking it was a search box.
56 comments
[ 3.1 ms ] story [ 119 ms ] threadSo apparently not.
Is there any way to do that in robots.txt? Someone did mention adjusting the crawl rate in Webmaster Tools -- would the author need to do this for every search engine?
Sounds like his issue is mainly with Google, not other search engines, so it’s not an O(n) problem he’s got to solve.
He removed the search before using a simple robots.txt...
I wouldn't call it an attack, but it seems more like some sort of autocompletion checking given how similar it is to the queries generated to Google if you use their search autocomplete feature --- one request for each character entered.
That said, Google's search quality has already taken a steep decline and I've been getting blocked very often for searching more obscure things now, so whatever "bot mitigations" they put in, I hope they don't make that even worse.
Me too! It seems like nearly any time I go past the second page for results they think I'm a bot.
- As a user, I type "foo.com" into my address bar. Having already gone to a specific page of that site before, autocomplete "helpfully" expands it to that page's URL "https://foo.com/this/is/a/page"
- It turns out I don't actually want that, I want the main page, so I start deleting characters until I'm left with "https://foo.com"
If Google took each of those deletions as a separate URL (either by bug or on purpose) and tried to verify said URL is valid, I could see a GoogleBot request being generated for each iteration between the initial URL and the full URL, starting with that initial one.
see:
https://www.hackerfactor.com/blog/index.php?/archives/678-Bo...
https://www.hackerfactor.com/blog/index.php?/archives/762-At...
https://www.hackerfactor.com/blog/index.php?/archives/775-Sc...
https://www.hackerfactor.com/blog/index.php?/archives/777-St...
https://www.hackerfactor.com/blog/index.php?/archives/779-Be...
I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails. Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something.
I can't help but wonder if this might be a similar case.
The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).
Or sensationalism. (it did land a top spot on HN)
[0] https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.
This is just normal background noise on the internet, one has to be pretty paranoid to assign any significance to it.
But it's weirder to claim that Googlebot is aggressively DDoSing you.
And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.
That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.
That's potentially a lot of money wasted on nothing.
> My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack.
And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.
That sounds insane. Is that actually true or is there an exaplanation of some sort?
As other comments mentioned, he could have easily prevented this with robots.txt:
- Is he sure it's Googlebot and not somebody spoofing headers?
- Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere?
- Is he sure it's the search crawler, and not some browser service trying to cache autocomplete combinations or results?
- Has he tried simple potential fixes like changing GET to POST for searches, or robots.txt to set a crawl interval to something like 30s?
It just seems so weird. Google crawlers have finite resources so this seems like a misunderstanding or really bizarre edge case or both.
It's not clear if he was getting hit by GoogleBot or e.g. FeedFetcher, which does NOT obey robots.txt, because it's triggered by human actions: https://support.google.com/webmasters/answer/178852?hl=en
There's also the possibility, as you say, that traffic was spoofed. Google publishes its netblocks over DNS, so it's easy to double check that.
In the author's previous post on the subject [0], he verified that the request were coming from Google IPs.
> Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere?
From the URLs quoted in the article, it looks more like some sort of (machine learning-y?) system trying to infer his site's canonical URL scheme.
[0] https://www.hackerfactor.com/blog/index.php?/archives/484-Go...
https://support.google.com/webmasters/answer/80553?hl=en
Then use robots.txt on any dynamic route like search.
A solid site map could help too.
http://www.datastreamer.io/
The main issues and complaints we run into are mostly unusual requests for indexing content.
People see the total number of requests and think it's a significant burden and just have an irrational need to protect their content.
It's almost a form of paranoia at some point.
I mean we have NO problem removing someone that doesn't want to be indexed and 99/100 we do so without a problem.
The only times we push back are when government organizations contact us as we feel we have a legal right to index the content. We haven't had to really fight over this because they usually agree and that's the end of the discussion.
Edit: add other; ambiguous.
I imagine what happened to this guy is that somebody coded up a site that used his as a service and google spidered it and went crazy trying to follow the links. It sucks, but that is what robots.txt is for. Anything that requires more than just serving a page should be blocked.
You can drive yourself insane reading your logs trying to guess what bots are trying to achieve. They seem to excel at spidering exactly what you don't want while ignoring the content that would actually be useful.
The one time I did have a problem with a misbehaving crawler was with something called 80legs[1]
[0] https://webmasters.googleblog.com/2011/11/get-post-and-safel...
[1] https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html
I'm sure their ML is pretty decent... but I do have to wonder if it's ever wound up leaving hundreds of comments on an anonymous forum or something, thinking it was a search box.
Google bot is stupid af and their support is even less helpful.
TL;DR: Google just scans you site even if it always gets 404s for millions of requests.
Google does field trials all the way in Chrome. Why would it be so unrealistic that they did similar trials with Googlebot?