I assume that some site has hostgator-related links with two slashes instead of one. Due to the two slashes, the GoogleBot doesn't realize that it's indexing their own results pages.
They probably want to index some pages on google.com, but not search results. To exclude search results, someone wrote something to exclude URLs that start with /search, and forgot that //search works the same way.
Even if you take the search string site:http://www.google.com//search and put it into a fresh Google search, it only returns HostGator coupons. Maybe someone from Google can explain it.
add -hostgator to the search query and you'll find best-seller-watches.com dominating the list. Add that one to your query and things get really strange.
Note that switching //search to /search eliminates the phenomenon.
Note too that all the results on page 1 and page 10 are related to hostgator and coupon codes. I expect that there is some site which contains some text or links that cause these results.
Note also that the `site:` search operator isn't supposed to include anything but a domain or subdomain: no http:// nor /search should be included.
Finally, note that the results are actually google search pages, though! So I do think this is some kind of bug.
But NOT an instance of Google indexing its result pages. Please change the title to 'This one weird google bug will make you scratch your head!' :)
Edit: andybalholm suggests (on this page) that the double slash is in fact causing the googlebot to visit those search results page and indeed index them. Hm, sounds true.
Has anybody visited the spamfodder pages and found instances of malformed yet operative links to google search? (I don't feel like visiting those sites on this machine on this network.)
I changed my tune at some point via seeing comments here. I posted a comment to that effect.
In hindsight, your comment alone would have changed my tune: nope, I can't explain the difference between a page appearing in search results and a page being indexed. Thanks for the illumination. :)
Does nobody here understand robots.txt? It's pretty easy to figure out what's going on if you do. I assumed most users here work with web technologies, but maybe the readership doesn't skew that way as much as I thought.
I think it might not be that they "index themselves" but they index links to google that others post on forums, it's common for people to link to "lmgtfy" so they probably index those links too. I don't see google "googling" on itself while indexing it's own searches. Unless Skynet.
The goal is to have an explicit Google search result which expresses the equivalent of "this Google search cannot be found via Google".
This will help construct a proof of Göogdel's Incompleteness Theorem.
Without being able to find anything in Google, including Google searches, and including that search for Google searches itself, Google is not a completely powerful search engine; however, it cannot be complete and consistent at the same time. There are searches which cannot be shown to be conclusively either in the index, or not in the index.
Many frameworks allow you to route URLs to actions instead of mapping to a file. I just tested it in one of my Symfony projects, and I was able to route /login and //login to two separate controllers.
Furthermore, it's pretty common to rewrite URLs, doing things like adding/removing trailing slashes, whatever. So it wouldn't be too difficult to have it condense multiple slashes into just one.
For example, this link worksfine:
google.com//////////////////////////////////search?q=foobar
Google search tries to cover a lot of typos or be pretty user-friendly for people who don't understand tech. I wouldn't be surprised if there's a grandma out there who thinks http://google.com//search is the correct method.
88 comments
[ 2.9 ms ] story [ 160 ms ] thread[1] https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.goog...
Any ideas why they're doing this?
https://www.google.com/webhp?gws_rd=ssl#safe=off&q=site:goog...
Note that switching //search to /search eliminates the phenomenon.
Note too that all the results on page 1 and page 10 are related to hostgator and coupon codes. I expect that there is some site which contains some text or links that cause these results.
Note also that the `site:` search operator isn't supposed to include anything but a domain or subdomain: no http:// nor /search should be included.
Finally, note that the results are actually google search pages, though! So I do think this is some kind of bug.
But NOT an instance of Google indexing its result pages. Please change the title to 'This one weird google bug will make you scratch your head!' :)
Edit: andybalholm suggests (on this page) that the double slash is in fact causing the googlebot to visit those search results page and indeed index them. Hm, sounds true.
Has anybody visited the spamfodder pages and found instances of malformed yet operative links to google search? (I don't feel like visiting those sites on this machine on this network.)
Correction: this works as you (or a muggle) might expect: https://www.google.com/search?q=site:https:%2F%2Fgithub.com%...
...Though logically the operator should be named `page:` now. :)
But that's clickbait! :)
That's what it looks like to me. Could you explain the difference?
In hindsight, your comment alone would have changed my tune: nope, I can't explain the difference between a page appearing in search results and a page being indexed. Thanks for the illumination. :)
google recommends the site:example.com/path shortcut itself https://support.google.com/webmasters/answer/35256?hl=en
and it's ok to use, as site:example.com inurl:path could mean example.com/hudriwudri/path, too
However, if you search site:http://www.google.com/search and show omitted search results, you get a bunch of results (all 404s).
If you do this there are some strange results on the last couple pages.
For example: Obama won't salute the flag | Phallectomy | horse+mating+video | feral+horses+induced+abortion | Lactating+dog+images | animal+mating+video | mating+mpg+-beastiality+-...
So, Half Life 3 confirmed.
But not http://www.google.com////search because that's just crazy, come on.
This will help construct a proof of Göogdel's Incompleteness Theorem.
Without being able to find anything in Google, including Google searches, and including that search for Google searches itself, Google is not a completely powerful search engine; however, it cannot be complete and consistent at the same time. There are searches which cannot be shown to be conclusively either in the index, or not in the index.
https://news.ycombinator.com//item?id=8297241
Where in all other cases tested it won't
Is this a server specific stuff? Or it's configurable
http://url.spec.whatwg.org//#concept-url-path http://www.nytimes.com///pages//politics//index.html http://www.bing.com////search?q=site%3Abing.com%2Fsearch%3Fq... https://www.cloudflare.com///index
Furthermore, it's pretty common to rewrite URLs, doing things like adding/removing trailing slashes, whatever. So it wouldn't be too difficult to have it condense multiple slashes into just one.
For example, this link worksfine: google.com//////////////////////////////////search?q=foobar
Google search tries to cover a lot of typos or be pretty user-friendly for people who don't understand tech. I wouldn't be surprised if there's a grandma out there who thinks http://google.com//search is the correct method.
https://www.google.ca/search?q=site%3Ahttp%3A%2F%2Fwww.googl...