HN Explorer

Hacker News bans Google and all other search engines (news.ycombinator.com)

318 points by jsrfded ↗ HN
Looks like news.ycombinator.com is rejecting all search engine robots with "User-Agent: * Disallow: /". This is unfortuante, I often do site: searches to find old threads here. esp since news.ycombinator.com doesn't have its own site search, this is the only way to find old threads.

pg, what's up?

128 comments

[ 3.4 ms ] story [ 184 ms ] thread
mikecane
What is the reason for this?
zitterbewegung
Probably to stop linkspam?
jsrfded
It's unfortunate because I often use site: to find old threads on ycombinator. especially since news.ycombinator.com doesn't have its own search facility, external search engines are the old way to comb through old posts here.
Frazzydee
That doesn't make sense. Why not just add rel=nofollow to all links?
[deleted]
(comment deleted)
drhodes
maybe to see what firms ignore robot.txt files?
idoh
My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of those pages crawled must generate a massive number of continuations to persist.

I know that this is one of the reasons that vote-links look the way they do.

If that's not the reason, then my #2 guess is that getting the whole site crawled is moving too many comments into the cache. If I remember correctly, whenever an uncached comment gets accessed, it gets cached (basically adding it to a hash), and thus added to the in-memory ram. That gradually raises the memory requirements for running the app, until pg does a restart, which resets the cache as well.

blasdel
And the regular restarts invalidate every single continuation-based URL
jgavris
very interesting
jgavris
the way i see it, they're going to have to put up a loginwall to stop crawlers...and then we'll just crawl back in anyway...
[deleted]
(comment deleted)
pierrefar
Does that also kill Search YC?

What about the "official" HNSearch (http://www.webmynd.com/html/hackernews.html )?

blasdel
The 'official' HNSearch is completely useless for anyone who would ever want to search HN. It's a Firefox Extension (!) that adds a 400px sidebar to your google search results pages showing site: searches along with an extra adsense ad. It's less useless than the 5 IE toolbars your annoying relatives live with, but just as obnoxious.

I just installed it again for a laugh, and it actually redirects to searchyc.com results when you click on its "Hacker News" header — even though pg staunchly refuses to ever link to searchyc. I made fun of him for it at his party before Startup School and he tried to laugh it off with "they're a YC company!" as if that weren't obvious on the face of it.

pierrefar
I don't use either. I'm interested for two reasons:

1. I use Google site: searches a LOT. I can remember reading about something I need now a long time ago on HN and I go find it.

2. One item on my "if I have time" projects list is to do "something" "fun" with HN submissions. Now I'm officially barred, regardless of what I would have thought up.

tptacek
What's pg's problem with SearchYC? It's incredibly useful.
jacquesm
That it competes with webmynd ?

It is a clear conflict of interest, HN should simply get the best search available, so the 'funded' webmynd guys should get off their butts and fix it or HN should support searchyc.

blasdel
He invited alaskamiller to that private party too — I was kicking myself later for not putting a little more fuel on that fire by pulling him into that conversation (not that I didn't piss off pg enough).
ddsmooth
That notwithstanding, if you go to Google.com, type hacker news into the search box and click "I'm Feeling Lucky" you will find yourself at a familiar web page. ;-)
jsrfded
Google indexes robots banned sites based on links. But notice that the snippet is gone. Google also won't be able to index HN posts and comments anymore.
tsally
So is SearchYC done, or do they use a scraper not a crawler?
grinich
What's the difference?
tsally
In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a website for updates periodically. Technically I suppose they all should probably respect robots.txt, but in my experience something you think of as a scraper rarely does.
Raphael
Well, what if I just saved every HTML page I visited. I'm clearly not a bot.
eel
What does this mean for http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies that no bots /should/ scrape the site.) Or will it be allowed?
qeorge
That was my immediate question too. In addition to searchyc, we lose every other HN mash-up present and future.

Maybe this will finally bring about an HN API.

weaksauce
robots.txt is really only like a note on the front door of your unlocked house that says "please only look at these items in these rooms." or "please do not look in my house" If someone wants to write a program to index the site they can still do so. Google, MSN, Bing, yahoo, et. al. will respect the robots.txt file though.
mkull
Or searchyc could simply ignore robots.txt
tkiley
Is this permanent?

HN is a good source of google juice for interesting new startups, and it would be a shame to see that go away...

qeorge
That's probably the main issue. HN's authority and DoFollow'd links make it a prime target for linkspam.

There's a lot of reasons I'm bummed about this, but I have to say: startups missing out on free PageRank is not one of them.

ericb
The links are nofollow'd now.

edit: Oh wait, seems that only comment links are nofollow'd.

weaksauce
Actually some of the links are nofollowed until it hits a karma threshold. Not sure of the value but it seems that a story with >50 votes is not currently nofollowed.
new_account
It seems that a lot is going on - in secret. Example: I just discovered that if I reload the page my votes for comments get rolled back.
blasdel
It wasn't rolled back: it was never counted.

There's a disenfranchisement heuristic that causes your votes to be recorded (can't vote again) but not counted. It usually persists for about a day. I think it's supposed to kill voting rings but its behavior is pretty odd.

The number in your browser gets incremented or decremented client-side independently of reality — it's doing a blind GET request by pre-fetching an 'image', not making an XHR POST and getting the new karma value in the response. Like a lot of things in Arc, it's just poor HTTP.

pierrefar
The way it's done now is that all new submissions are nofollowed until they hit a certain karma threshold. A while back when I looked at this, the threhold was about 3-5 up votes.

This is actually a very neat and simple way to exercise editorial control to remove the nofollow: if enough people in the community like it, there is something useful about it and it is vetted.

chaosmachine
If this is an anti-spam measure, I expect it will be about as effective as "no follow" was. There's still plenty of good reasons for spammers to submit crap. RSS and Twitter syndication of links, for example.

On the other hand, if the goal is to push HN back into semi-obscurity by making it harder to find, it might work.

g0atbutt
That's pretty disappointing. I'd love to hear an official explanation of why this choice was made.
cookiecaper
I quite agree. I use Google to search HN. I just used it the other day, because I didn't remember this one guy's startup or username, but I knew it involved family websites. I was able to find it after a bit of search-fu. I am sad to see that HN won't be indexed by Google anymore. :(
dantheman
One way to search news.ycombinator is to visit searchyc.com
JeffJenkins
Unless they're obeying robots.txt, which they probably should be
jxcole
I tried search google go with and without quotes and the results were the same. Also, very few of the results actually pretained to google go. I would still rather have google...
joe_the_user
Yes, the way that references vanish from hn, I really need Google to find things a couple of days old. I have tried and found other search methods not worthwhile.

Grumble...

Qz
Hmm, does search-fu qualify as something you can put on a resume?
zachbeane
I wonder if part of the reason is that the Arc web server doesn't send HTTP status codes properly. Every URL path, valid or not, returns a 200, even if the content is only "Unknown."

For example, check the headers sent back for:

http://news.ycombinator.com/no/way/this/is/possibly/valid

    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Connection: close
If a search engine sees that, it will periodically revisit it forever.
pierrefar
The correct solution is to fix the Arc server to return correct HTTP headers, not block everything using robots.txt.
Qz
Maybe they're blocking everything while they're in the process of fixing it.
Xichekolas
That may be the correct solution, but I'm sure the current option was chosen more because it was the quick solution.
sjs
Until someone creates an HTML link to such a document, how does this problem manifest itself?
zachbeane
The site itself created zillions of them itself with its random, expiring fnids. Although the scheme has changed now, cralwers would still hit those obsolete URLs.
pvg
HN is very, very slow. Removing crawlers likely helps with that.
tjic
Removing a leg or two could also help one towards a weight loss goal.
dmoney
But then you lose a lot of options for exercise. Walking and running are definitely out.
lsc
I would bet good money that it's nothing that a good dedicated server with, say, 32GiB ram and 2 or 4 sas spindles couldn't fix.
pjhyett
You can set gbot's crawler rate with google webmaster tools if that's the issue.
Apreche
This sucks for one major reason. Sometimes I want to dig up a link that was on HN, and I don't know where else to find it. So I search google with site:ycombinator.com. If they block Google, then that won't work.

At the very least, add a search engine to HN itself so I can search for old links.

Chirael
Agreed 100%

Why produce a great site and then not let your users access its valuable content?

Things scroll by the front page so fast it's not really fair not to provide a search feature.

I don't care if they let external search engines (Google et al) index it, but they should at least provide /some/ kind of internal search feature themselves.

IMO providing a search feature is probably at least as effective, if not more so, than the "noprocrast" flag.

After all, if I know I can search past items at any point in the future, I don't feel a pressure to load up the front page several times a day.

Whereas if the fast-scrolling front page is the only way to access items, it encourages that sort of rat-hitting-the-bar behavior I learned about in college psych classes ;)

LouCypher
The robots.txt file's content-type is text/html, not text/plain. It won't work. I'm still able to access the contents of this site from Yahoo Pipes. Don't worry about it.
simonw
I noticed that the other day while playing around with the YQL console - YQL obeys robots.txt, so the Hacker News data table doesn't work any more.
[deleted]
(comment deleted)
petercooper
Will this break Google Reader and other RSS readers from legitimately using the RSS feed here? After all, Google Reader uses a bot to read the feed and allows us to search it from within their app.. much like their search engine does with regular pages :-) (That is, "web robots" aren't just spiders.)

Also things like http://hacker-newspaper.gilesb.com/ and http://hnsort.com/ become less legitimate due to this. If the reason is to reduce the SEO benefits of getting a link on HN, just "nofollow" everything instead..

(Update: Googling on this topic brought up a page of my own where a Google Reader engineer explained how Google Reader deals with robots.txt - http://www.petercooper.co.uk/google-reader-ignores-robottxt-... - though their definition of Web robot is far from universal)

tptacek
I'm interested in the justification for this, but I'm happy about it. I'm actually uncomfortable with how high Hacker News comments score on Google.
chaosmachine
start posting your comments in rot13? ;)
earle
Some Brainfuck to implement this suggestion:

,[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>>>+++++[<<----->>-]<<<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>>>+++++[<<----->>-]<<<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]>>.[-]<<,]

jamesbressi
Please enlighten me as to why you are "uncomfortable with how high Hacker News comments score on Google." I mean it sincerely, I'd like to know because I have no clue why that is an issue?
tptacek
Talking to people in a relatively small forum like Hacker News: totally comfortable.

Talking with the entire world by having my comments associated in ways I don't expect with keywords: less comfortable.

jamesbressi
Thank you and that makes total sense. Point and case?:

I googled an executive of a company that was looking to do business with me. I came across some google groups discussions that he was part of from a couple years ago... and let's just say the material would be a dream for someone with a vendetta against the person.

Thanks for the perspective!

proee
This makes me sad. I often do site:news.ycombinator.com google searches for key topics.
Mc_Big_G
I do exactly the same thing at least once a day. I can always remember just a snippet of the post or subject I want to find and that type of query always gets me what I want. I just used it today to find the YC company who is doing the journalism stuff so I could send the link to my nephew.
Concours
That's my standard method as well to search HN.
andrewcooke
surprised no-one has suggested that this is to improve performance. this place can be very slow sometimes, and blocking robots can reduce load.
keltex
That's not a legitimate reason to ban google. You can always (using google's webmaster tools) throttle their indexing. Or you can use search-engine specific lines in robots.txt (Bing anyone?) to ban misbehaving bots.

The only reason to do this is if you don't want your site indexed by google. Which I really can't think of a legitimate reason to do so.

andrewcooke
true, but it's a busy time for yc. maybe this was just a stop-gap measure? when did it start?
[deleted]
(comment deleted)
jonknee
... And you can speed the site up. There's no reason HN has to be so slow.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
WalkingDead
I guess that might have to do with load these search engines generate. Previosly in a thread we checked that comments posted in HN appear in google search after a minute of posting. That should create a good amount of load on HN. Multiply it for all the search engines in the wild, and PG probably have decided blocking those woun't do any harm. My guesss. Could be wrong, or it might be temporary or a mistake.
redsymbol
Everyone RELAX. There's probably a good explanation for this, maybe it was even an (easily correctable) accident. Give Paul a chance to respond before raising the pitchforks, okay?
prs
Truth be told: On various occasions the site: operator on Google came in handy for me to dig out some nugget of information from the archives of HN.

I can't await to hear the reason behind that decision.

python123
This is horrible. Now what's a talentless, middle-aged career software engineer with hopeless dreams of entrepreneurship going to do with his time?
ars
The robots page is returning the mime-type text/html instead of text/plain
sahaj
i just googled this: http://www.google.com/search?q=site:news.ycombinator.com+Hac...

and found this: http://news.ycombinator.com/item?id=165279

one of the top replies:

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

borism
ugh

perhaps smart hackers shouldn't block everything out like this without any discretion whatsoever?

Matt_Cutts
Does anyone know why HN decided to disallow search engines? It's certainly up to the website owner to decide, but there are good ways to (say) reduce the load on the web server without blocking search engines entirely.
[deleted]
(comment deleted)
seiji
The HN code keeps everything in memory (lazily loaded from disk). If every search engine and me-too crawler out there hits every page, you constantly have the entire HN corpus in memory. I assume the memory overload is what leads to the site being slow and unreliable. I don't know if the arc code has any LRU scheme or if everything sticks around until the server falls over.

HN just needs a super simple caching proxy in front of it to reduce load on the app server. Alternatively, just generate static pages for all topics older than 5 days.

Will sysadmin for YC dinner invites.

euroclydon
Didn't I read here that the MSN bot refuses to obey the robots.txt file? Maybe I'll have to search HN with it.