I wasn't really saying anything about hndex, apart from that it indexes the contents of external articles submitted to HN, as you said.
The correction I was making was about HN Algolia search (which is linked from the search box at the bottom of each HN page), which only indexes content on news.ycombinator.com itself – i.e., article titles, comments and text-only posts like Show/Ask HN – but not the external content in submitted articles.
This is really useful! You might want to consider hiding the “More”-button if the current page isn’t filled up, so as to not just have an empty page when clicking it.
For some reason I couldn’t find my own blog post[0], even when searching for the embarrassing typo I made in the title - acommodating.
Very nice. Works as expected.
I stumbled about two things that could be improved:
- Add a search Button for convenient mobile use or if a user copy pasts things into to the search field using the mouse
- Add a comment counter on the result page, Since you index every article a lot of them have none or very few comments.
Oh and just a warning, depending on the jurisdiction providing the cache could be problematic under copyright laws since its basically a copy of the article.
Ohhhh I first thought this was a search engine for hacker news and I was wondering why the hell anyone would reimplement this, but this is fricking cool. Tbh I wanted to implement something similar for more than hackernews, but this is a nice thing to have. I especially love that you implemented a cache :)
Seems nice. Sort and filter functionality would probably add to this, but I will bookmark this for sure and try it out as a search engine for tech topics in general.
Looks like this search engine searches the _articles_ and links that have been shared, whereas hn.algolia.com only searches title, author, and the text of a post if it is a text post.
Neat project. It’s a bit depressing how many of the article links don’t work anymore. For example, only the official signal vs noise article works on the first page of results for Basecamp: https://hndex.org/?q=Basecamp
How does ranking work? Apologies if I’ve missed the explanation.
They easily could (I think it would take the average HN user no more than a couple of hours to implement this functionality), but they won't, because they're very reluctant to make changes.
This is also why HN still looks like a site from the 90s, instead of New Reddit (thank God).
It’s not all bad. We still have HN as it is now. Status quo is its own kind of success. HN is essentially feature complete, so I don’t blame them for not trying to fix what ain’t broke.
Articles which have been posted to HN often end up in the archives of Archive.org and Archive.is in my experience.
The OP site also has a “cached” link for each article, don’t know if you saw that.
Also, to the maker of HNdex, consider adding links to Archive.org and Archive.is next to the cached link you have, so that readers can click through and check if they have a version of it in case there’s images etc
That's awesome! I searched for MuleSoft, not expecting anything in particular, and found a random fact (at the end) that certainly wasn't in the title of the article.
Bug report: in Safari on dark mode, the text in the search box is almost the same color as the background (white on white).
I don't know why but to me the background is #000 black with #fff text by default...
It's such an intense "dark mode" that I couldn't read the first full article of interest I found because the contrast was killing me, and then coming back to regular HN left me nearly blinded for a few secs
I love this. The search box had an option to autocomplete terms I had recently searched, I clicked on one (a term I was interested in, obviously) and immediately found an interesting HackerNews post. Nice!
I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. This seems like a great way to uncover some of the latent value of old HackerNews content.
Is there a way to suggest content too? e.g. If you liked that, you'd probably also like X, Y, and Z.
>I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff.
With Reddit and HackerNews, I want a relative ranking index. I can search by the top content, but something 5th today could have more votes than the top submission of 2015 because of forum/subreddit growth.
I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day.
> I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day.
Yeah, this is definitely possible with public data. I did something similar on reveddit [1] for removed reddit content. Hovering over the graph shows the item with the highest vote ratio [3], and clicking skips to that point in time. Code is here [2] for anyone interested and I apologize in advance..
Absolutely. HN user 'EvanMiller had a lot to say about this, 11 years ago. His tl;dr is that the ranking score should be "the Lower bound of Wilson score confidence interval for a Bernoulli parameter."
I believe HN's ranking system is extremely creative and works great for the main page and day-to-day use (my understanding is that it additionally makes use of time-decay terms for comments and stories). Like you said, it's really just historical search (i.e. algolia) that seems broken.
Agreed! The problem I see is that the score does not correlate well with the perceived quality of the community. I'm researching that for some time now and am in the process of preparing a blog article with data analysis and solution approaches.
Yes, with so much context missing from the results it's pretty confusing.
Ideally for this to be useful I'd want date on HN, date of article (although I see that'd perhaps be hard to extract from unstructured pages) and also the HN points, as those are a major proxy to quality usually.
As others have said it has a nice crisp UI and i like that it's so quick.
This produces such a unique set of results for things.
Fantastic project and well worth creating!
As another commentor has noted - this totally disobeys recency bias and throws up interesting articles for a topic.
Edit - it is disappointing how many of these links 404. But even if that's the case the headline and intro is a set of time capsules of sorts nonetheless.
Very cool implementation. I am wondering have you considered using something like https://github.com/cliqz-oss/adblocker which can sit on top of headless browser and do not require bridging to an extension.
I would love to see the source and/or a write up about the stack used here. I've thought about making something similar, but got distracted while in the planning phase.
Great idea, I appreciate how fast it returns results! Just needs more more control over the search parameters and figuring out why articles like the example I posted above aren't working and you got yourself a nice HN search.
78 comments
[ 4.8 ms ] story [ 162 ms ] threadMost articles have no comments. Yet one can search on hndex for terms only found in the article, not in the title.
The correction I was making was about HN Algolia search (which is linked from the search box at the bottom of each HN page), which only indexes content on news.ycombinator.com itself – i.e., article titles, comments and text-only posts like Show/Ask HN – but not the external content in submitted articles.
For some reason I couldn’t find my own blog post[0], even when searching for the embarrassing typo I made in the title - acommodating.
[0] https://news.ycombinator.com/item?id=23426951
- Add a search Button for convenient mobile use or if a user copy pasts things into to the search field using the mouse
- Add a comment counter on the result page, Since you index every article a lot of them have none or very few comments.
Oh and just a warning, depending on the jurisdiction providing the cache could be problematic under copyright laws since its basically a copy of the article.
Seems nice. Sort and filter functionality would probably add to this, but I will bookmark this for sure and try it out as a search engine for tech topics in general.
Thanks for creating this!
(Algolia was backed by YC btw)
How does ranking work? Apologies if I’ve missed the explanation.
> How does ranking work?
Great question, it's something I'd love to know more about as well.
This is also why HN still looks like a site from the 90s, instead of New Reddit (thank God).
The OP site also has a “cached” link for each article, don’t know if you saw that.
Also, to the maker of HNdex, consider adding links to Archive.org and Archive.is next to the cached link you have, so that readers can click through and check if they have a version of it in case there’s images etc
HN feature request: Add submissions to archive.org (or equiv), include that cache link with story.
Bug report: in Safari on dark mode, the text in the search box is almost the same color as the background (white on white).
From "GitHub was also talking to Google about a deal, but went with Microsoft instead" https://www.cnbc.com/2018/06/05/github-interest-from-google-... - Salesforce got MuleSoft for $7.5 billion, Microsoft got GitHub for $7.5 billion.
It's such an intense "dark mode" that I couldn't read the first full article of interest I found because the contrast was killing me, and then coming back to regular HN left me nearly blinded for a few secs
- What is your stack?
- How often is your database updated?
- And will it be open source in the future?
I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. This seems like a great way to uncover some of the latent value of old HackerNews content.
Is there a way to suggest content too? e.g. If you liked that, you'd probably also like X, Y, and Z.
With Reddit and HackerNews, I want a relative ranking index. I can search by the top content, but something 5th today could have more votes than the top submission of 2015 because of forum/subreddit growth.
I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day.
Yeah, this is definitely possible with public data. I did something similar on reveddit [1] for removed reddit content. Hovering over the graph shows the item with the highest vote ratio [3], and clicking skips to that point in time. Code is here [2] for anyone interested and I apologize in advance..
[1] https://i.imgur.com/p3Bi5IS.png
[2] https://github.com/reveddit/ragger/blob/master/revddit_aggre...
[3] https://www.reveddit.com/r/worldnews/?showFilters=true
I believe HN's ranking system is extremely creative and works great for the main page and day-to-day use (my understanding is that it additionally makes use of time-decay terms for comments and stories). Like you said, it's really just historical search (i.e. algolia) that seems broken.
https://www.evanmiller.org/how-not-to-sort-by-average-rating...
https://news.ycombinator.com/item?id=15709405
https://www.evanmiller.org/bayesian-average-ratings.html
https://www.evanmiller.org/ranking-items-with-star-ratings.h...
https://redditblog.com/2009/10/15/reddits-new-comment-sortin...
Work in progress:
https://github.com/fdietze/downvote-scoring
https://felix.unote.io/hacker-news-scores
Ideally for this to be useful I'd want date on HN, date of article (although I see that'd perhaps be hard to extract from unstructured pages) and also the HN points, as those are a major proxy to quality usually.
As others have said it has a nice crisp UI and i like that it's so quick.
Fantastic project and well worth creating!
As another commentor has noted - this totally disobeys recency bias and throws up interesting articles for a topic.
Edit - it is disappointing how many of these links 404. But even if that's the case the headline and intro is a set of time capsules of sorts nonetheless.
Go back and click "cached".
That was a cool experience.
I often find myself searching HN for opinions on different technologies over time and this will be invaluable for that use case.
My only wish is that it should be possible to sort chronologically.
http://kakapo.susa.net:8080/cfs/
I abandoned it when Google deprecated the WebRequest API, but the code's still available on GitLab. https://gitlab.com/ksangeelee/cfs_build
It allows article score and uBlock Origin 'hits' as search criteria.
Great idea, I appreciate how fast it returns results! Just needs more more control over the search parameters and figuring out why articles like the example I posted above aren't working and you got yourself a nice HN search.
[1] https://news.ycombinator.com/item?id=22830472
Is it possible to give a little inside peek of the techniques, algorithms and tech stack etc. you have used?