MetaFilter has been doing this for years too and it works ridiculously well. You have to see a list of potential matches before you can commit your post. So +1 to that.
People won't use it. Decent search has been available via SearchYC for years, and no one seemed to use it to prevent duplicate and/or similar submissions. Further, if you submit via the bookmarklet there is a huge incentive not to search.
The feature that jumps out at me is that I can see comment karma scores for all the comments that I search up with a keyword search. Evidently the upvote/downvote buttons I see next to the comments in the search results don't change the comment karma scores.
I noticed the same thing, especially for current discussions. But then it changed. It looks like there's now a 5-day threshold for karma point exposure.
HNSearch consists of a simple javascript webapp hosted at hnsearch.com which sends ajax requests directly to ThriftDB. ThriftDB returns items in the search response which makes it ideal for this type of architecture. For a more detailed explanation checkout the API docs:
We wrote a crawler to download data from HN, parse it into JSON, and upload it to ThriftDB. The webapp at hnsearch.com sends requests directly to ThriftDB. There's a ~15min lag between HN and HNSearch.
Should be a good test of the ThriftDB technology. Too bad the HN discussion about ThriftDB didn't come up in my test search, found it via Google instead.
Do you typically browse with JavaScript disabled? I would think that these days it would seriously change the browsing experience and many sites would be unusable. Just curious. :)
I have JavaScript disabled by default, and selectively enable it for the sites where I feel it is worthwhile. I'm a longtime NoScript user, and these days I use NotScripts with Chrome.
I search HN a lot, so am excited by this native implementation. I would be curious to know more about how the "relevance" sorting algorithm works though. When I did a test search for "domain registrar," for example, the top result was a comment with a score of -4. It seems like there are many ways to implement that feature, so would would be quite interested to hear more if the creators were able to share some thoughts on the general "relevance" problem.
We're using the HN hotness algorithm with some points boosts to surface older items. For a full explanation of the ranking algorithm check out the API docs:
I like hnsearch a lot, but I'd like us to take a second to thank whoever was running SearchYC, which for the past couple years has been practically indispensable in keeping up with this community.
SearchYC was also more fully featured, with lists of top users based on points per submission, points per comment, etc. - at least until pg removed comment points from the site.
Just off the top of my head here are some unanswered questions:
Why did Comcast shut them down? why did comcast have the authority to shut them down? Why didn't the searchyc admins take their site somewhere else? Why have they said nothing here on hn? Why were they shutdown mere days before pg announces an official search (hopefully just a coincidence). What happened to their data (it's especially valuable now that we can no longer see comment scores). Etc.
No outrage intended sorry if it came across that way.
I was commenting on the abrupt departure of searchyc not on the new search. Haven't seen the new search yet since it's not working on my blackberry for some reason. And I don't have my laptop with me.
"Hopefully it's just a coincidence" that SearchYC lost its Comcast connectivity right before Paul Graham announced HNSearch. No. I don't think I misread the undertone.
Regardless, we don't need to beat this to death. Have a good weekend.
any chance of integrating the search box into the top bar?
i use some auto pagerize plugins that sort of get in the way. i know i could just turn it off, but that would brake its purpose.
also, i suppose this would help with general visibility of the search box, as most sites have theirs on top of their pages!
but it's definitely great to see search build into the site!
will make finding stuff on hn a lot more convenient.
You can do the same thing with a keyword bookmark in Firefox.
I've been searching HN all this time with a custom Google search including "site:news.ycombinator.com". I'll have to see if this is better. Google offers results filtered by date, which is really nice, since technology moves so incredibly fast.
You can add HNSearch as a custom search engine in Firefox as well. Just visit hnsearch.com, click on the favicon in the browser's search box and select "Add HNSearch".
I can't reproduce the Firefox steps you mention. I click on the favicon in the browser's search box and I don't get a chance to select "Add HNSearch," but rather a message saying "This website does not supply identity information."
Or, by "id"? I looked, but I wasn't able to figure out, for example, how I'd get pg's top-level comment here from the id "2619846". I was able to find it with this search, but it requires a second-part to the "_id" that I don't know where to find: http://api.thriftdb.com/api.hnsearch.com/items/_search?q=... (Trying to filter for "id" instead of "_id" throws an error, for some reason.)
Well that explains why I failed at fetching an item by id, and I tried a whole bunch of different ways. You should really mention somewhere that the "id" referenced is an internal id, and not the HN id.
For example what's the difference between "filter[queries][]" and "q" arguments for http://www.thriftdb.com/documentation/rest-api/search-api? The way I understand, q does a simple keyword matching on all fields while "filter[queries][]" is more like SQL "where" query. In that case, what operators are supported?
The `q` argument is used to perform a full-text search across all fields. One benefit to using `q` is that you can rank results based on field matches (using the `weights` argument).
The `filter` arguments are used to cut the data before performing a full-text query:
`filter[fields][fieldname]` can be used to filter on fieldnames:
However - I don't always want to upvote a post just because I want to check the discussion. Published this with an assumption that I'm not the only one.
Pardon me and I don't want to come off as ignorant, what is the big deal with this? I have been using Google to search hackernews articles by querying `<subject> site:news.ycombinator.com` and it has worked wonders.
Since hackernews is all publicly available, Googlebots must have indexed every single page of this site. And we all know how good Google has been with ranking and relevance. Why did HN decide to reinvent the wheel? Why didn't HN use the Google Custom Search plugin?
The problem with Google Search for a site like Hacker News is that it doesn't have the same level of understanding of the metadata that makes up the site. A good example is sort-by-date (which Google can approximate based on the date something was first spotted by its crawlers, but it's not nearly as accurate as having access to the "date" field in the underlying data structures) - another is "just search comments by this username".
HNSearch is an Octopart/ThriftDB project to test out our search technology and give back to the HN community.
Google has access to the link structure of the internet so it's great for macro searches. However, if you want to do a site-specific micro search then it usually helps to have access to the underlying metadata (e.g. points, karma, timestamps).
SearchYC allowed results to be sorted by date of submission, allowed searches by username and points, and other aspects that Google simply doesn't understand and can't provide.
128 comments
[ 3.1 ms ] story [ 200 ms ] threadhttp://www.hnsearch.com/api
And here's a sample search response from ThriftDB:
http://api.thriftdb.com/api.hnsearch.com/items/_search?q=fac...
http://news.ycombinator.com/item?id=2581652
These results are as of 1pm EST on Saturday, as this very comment thread gets indexed the ordering may change...
That's still a great thing to have. Simple and efficient.
http://www.hnsearch.com/api
Email hn@dotcloud.com to get started.
It describes an action, but explains nothing.
Just off the top of my head here are some unanswered questions: Why did Comcast shut them down? why did comcast have the authority to shut them down? Why didn't the searchyc admins take their site somewhere else? Why have they said nothing here on hn? Why were they shutdown mere days before pg announces an official search (hopefully just a coincidence). What happened to their data (it's especially valuable now that we can no longer see comment scores). Etc.
You obviously haven't even looked at HNSearch if your big concern is comment scores.
Sheesh!
I was commenting on the abrupt departure of searchyc not on the new search. Haven't seen the new search yet since it's not working on my blackberry for some reason. And I don't have my laptop with me.
Regardless, we don't need to beat this to death. Have a good weekend.
http://news.ycombinator.com/item?id=2621391
It would be a fun challenge to see who can come up with the best and most useful solution :)
For the next two weeks there's even an API contest going on! http://www.hnsearch.com/contest
but it's definitely great to see search build into the site! will make finding stuff on hn a lot more convenient.
Add a new search engine -> Hacker News
Keyword -> hn
Url with %s in place of query -> http://www.hnsearch.com/search#request/all&q=%s
You can do the same thing with a keyword bookmark in Firefox.
I've been searching HN all this time with a custom Google search including "site:news.ycombinator.com". I'll have to see if this is better. Google offers results filtered by date, which is really nice, since technology moves so incredibly fast.
For the Chrom custom search engine it would be better to use the non-javascript url: http://www.hnsearch.com/search?q=
That will get redirected automatically and won't change with the webapp implementation.
You can filter by date/points with HNSearch.
http://www.hnsearch.com/api
which should be a lot faster than crawling HN itself to get the same data.
Edit: Nevermind, figured it out: http://api.thriftdb.com/api.hnsearch.com/items/_search?prett...
The `id` attribute on the other hand is an item's submission id. It's not indexed though so you can't do lookup by id.
How do I find this comment through the API? How do I find its replies?
I would like to lookup items by HN id.
The `filter` arguments are used to cut the data before performing a full-text query:
`filter[fields][fieldname]` can be used to filter on fieldnames:
http://api.thriftdb.com/api.hnsearch.com/items/_search?filte...
`filter[queries][]` can be used to add arbitrary filters:
http://api.thriftdb.com/api.hnsearch.com/items/_search?filte...
http://api.thriftdb.com/api.hnsearch.com/items/_search?filte...
The syntax for filter queries is the same as solr:
http://wiki.apache.org/solr/CommonQueryParameters#fq
http://kaspa.rs/hn-bookmarklet/
If the current URL is added, it will go straight to discussion thread (without voting). If not, it will ask if you want to submit it.
However - I don't always want to upvote a post just because I want to check the discussion. Published this with an assumption that I'm not the only one.
When I build this web app http://www.vcarrer.com/2010/11/hacker-news-mobile-front-page... I need to use Yahoo YQL to obtain JSONP.
So pretty please, add JSONP.
http://www.thriftdb.com/documentation/rest-api
Since hackernews is all publicly available, Googlebots must have indexed every single page of this site. And we all know how good Google has been with ranking and relevance. Why did HN decide to reinvent the wheel? Why didn't HN use the Google Custom Search plugin?
(I am just curious to know and not criticizing.)
Google has access to the link structure of the internet so it's great for macro searches. However, if you want to do a site-specific micro search then it usually helps to have access to the underlying metadata (e.g. points, karma, timestamps).