HN Explorer

Ask HN: What to do when Google ignores you?

2 points by Frozenlock ↗ HN
Hello gentlemen,

I've spent months building my tools and services. I'm actively trying to promote my website, but Google stopped crawling my site for 3 months now. I've checked all my commits around the time Google stopped crawling, but there's nothing there that should have an influence!

I tested my website with proxies, web-sniffers, users from around the world: the website works for ALL of them. However if I try to 'fetch as Google' in the Google's webmaster tools, it always results in 'page unavailable'.

I emailed Google, but all I received (even after multiple tries) was pre-made answers of this kind "Have you checked our forums?" YES, I did! I posted my question, and didn't get any answer, which is why I'm emailing you now!

So yeah, I'm happy to see my website on DuckDuckGo, but I would really like to see it on Google; especially considering that my customers aren't necessary cutting edge...

- I tested with multiple proxies and web-sniffer tools. - I checked my DNS - I checked if my ISP was blocking Googlebot IPs - I even checked with Wireshark and can see the Googlebot connection initiated!!

I asked Google to get a more useful explanation than just 'page unavailable', but alas...

So now I'm asking help from you. Here is my website: http://goo.gl/aSX2u Is there anything obvious that I'm missing? Why is Google ignoring me?

Thank you very much in advance!

14 comments

[ 2.0 ms ] story [ 42.9 ms ] thread
rcavezza
Do you use Google webmaster tools? They should show potential reasons on there and which pages are currently indexed.
Frozenlock
The reason shown in the crawl error is "Robots.txt fetch".

This isn't very surprising as 'fetch as Google' returns a page unavailable for everything.

(It's my understanding that a 404 on robots.txt is OK)

27182818284
In the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?
Frozenlock
I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.
AznHisoka
Try changing from HTTPs to HTTP temporarily (a week or so) and see if that helps.

Another solution: Come up with a new domain, do a redirect.

Frozenlock
I did try without HTTPS, and even without GZIP. Not for a week, but enough to test the 'fetch as Google' option in the Google webmaster tools.

I've also read that Googlebot shouldn't have any problems with HTTPS.

AznHisoka
Google claims they don't have problems all the time. Doesn't mean you should trust them. Give it a shot, what will it hurt?
Frozenlock
Tried again without HTTPS and this time it worked (What the f..).

I hope this will be fixed soon, I would prefer to force HTTPS everywhere.

Anyhow, thank you very much, at least now the bot will be able to crawl my site!

propercoil
three problems off the bat: 1. you have only 10 backlinks from different root domains. 2. one of them is bacnetwiki.com (yours). This can signal a backlink pyramid. 3. your server is dropping requests when pinging from holland. Something is definitely wrong here
Frozenlock
Thanks! The pinging is deactivated on my router, no surprise there.

The problem is not the ranking, which backlinks influence, but the ability of the Googlebot to crawl the site.

Also, I did not add the entry to BACnetWiki. In fact it's probably one of the only mention of the site that I did not made. :-)

orangethirty
I'm going to point the Nuuton crawler towards your site. Shoot me an email to give you results.
Frozenlock
Thank you!

You can send me a mail at frozenlock@<the-domain-name>.com