Show HN: Search HN for interesting comment sections (searchhacker.news)
I built this tool to help me find interesting discussions on Hacker News. I love reading HN discussions almost more than the articles themselves. However, I found that full text search, although highly performant, is not always good at surfacing interesting discussions on a certain topic -- especially if you don't know what to search for exactly.
I built this by scraping the most recent ~6 million posts (that's about 2 years of history) and putting the resulting posts and their vector embeddings into Postgres.
Let me know what could be improved, and if you'd like a more detailed writeup of how this was built :)
12 comments
[ 5.0 ms ] story [ 39.7 ms ] threadThe sample searches were particularly strong for uncovering discussions on specific subjects, i.e. 'Dedicated Vector Databases'.
With less focused searches, such as 'Deep Work', I notice some results feel only loosely related. Is this an aim – to broaden the scope of discovery that may not seem directly related?
It's all very serendipity-driven, and this is mostly a tool for wasting time, but you might discover some interesting (and hopefully on point) conversations this way.
Anyway, valuable feedback. Maybe in a next interation, I'll always do both searches and merge the resulting ranking.
Also, somehow this has a relatively big horizontal margin on mobile.
I see what you mean about the margin on mobile... note that I'm one of the most unqualified front end people I know and this was mostly done using github copilot, I might not be brainy enough to understand the intricacies of css responsive layouting ;)
I'll post a writeup sometime too!
Interesting project, though. As a shortcut, I usually just use the hidden /active URL to find the interesting stories. Generally these are the ones with the most active commentary and often they get pushed off the main HN screen quickly by the "flamewar detector". So you have to be quick to find them, or just use /active.
Interesting aside on flamewar detection: you can visualize the comment embeddings in a 2d scatterplot, color coded by username. As you'd expect, a great deal of dang's comments occupy a distinct position in comment-space.
Three of examples on the current home page surface some toxic threads (“llm waifus”, “internet of shit”, and “AI Doomers”)… which while controversial aren’t as rich or insightful as say the comments from people that built Rust, the whole earth catalog, or the first x86 chip.
[1] https://news.ycombinator.com/bestcomments