HN Explorer

Show HN: Hacker News with categories and popularity (newshack.io)

149 points by federkasten ↗ HN
Did you want to know topics of the news or popularity of posts? "Hacker News Hack" helps you to find news based on some categories and importance or popularity.

It tagged news with 9 categories with a natural language processing technology and supervised learning for categorizing news.

It also calculates a speed of number of comments in each posts as popularity based on term, that is 1 hour, 1 day, 1 week, 1 month, or 1 year.

67 comments

[ 4.3 ms ] story [ 135 ms ] thread
joaomsa
Very nice. How you determine categories?
federkasten
Using natural language processing and supervised learning, it automatically tagged news with 9 categories.
xux
That's really interesting. Will you open-source the NLP algorithm?
federkasten
I am planning to open source it in several months. (Our codes have not been well-commented and well-structured yet...

Our implementation and algorithm detail is followings.

Its categorizing process is written in Python.

Using nltk, it makes corpus with TFIDF model from HN topics and comments. And it generates classifiers from this corpus with SVM algorithm using scipy and numpy.

FYI, its web interface is written in Clojure and ClojureScript.

hnriot
presumably you've trained it with hand annotated content, or bootstrapped from a few choice hn searches (like ?q=jquery will give you a web tech category)
federkasten
Yes. You are right.

I trained classifiers with hand annotations (about 1000 contents or so)

Houshalter
I wonder if you clustered HN articles to find categories automatically. That may not be useful, but I'd be interested in seeing how HN articles are naturally categorized.
TrainedMonkey
Awesome! Too bad jokes category does not have a lot of content, pity.
NAFV_P
I think something needs to be done about that.

But, humour is almost universally subjective. There are some aspects of Obama-Care that I thought were highly amusing, especially the "500 million lines of code" gag.

Here is a very serious article about the C programming language.

http://uncyclopedia.wikia.com/wiki/C_programming_language

deevus
"Lisp - C for parentheses who want to meet other parentheses."
colept
Too bad this isn't a site to find jokes - and yet more and more people want it to be like reddit.
NAFV_P
I think only certain types of humour are suitable for this site, geared more to the interests of HN readers.

Having a giggle does not have to be a frivolous activity.

cooper12
View the jokes over a year. There are 88 results. Not neglible IMO.
ozh
wow that's a joke every 4 days! Who knew we were having so much fun.
GigabyteCoin
Pretty neat!

FYI: I just noticed your time calculations aren't working. Search for "item.time_str" on your indexes.

[deleted]
(comment deleted)
dclara
This is interesting. But it's not quite sync with the News on the first page. What do you sort by? If it works well, I'd like to use it.
federkasten
Thanks.

It crawls HN topics every one hour. So news in our application would delay up to one hour.

jkarneges
If you're hitting HN's access thresholds, you might consider incrementally parsing data from http://hnstream.com

That may help with analyzing comment content at least. Not so much with news rankings though.

federkasten
Thanks. hnstream is really cool.

You're right. I have limited frequency of crawling due to HN's access thresholds.

hnstream will help me. I will try it.

tehaaron
Looks awful on mobile would be great if you could make it responsive.
federkasten
Yes. I will support responsive design in the near future.
monsterix
This is super useful!

I have been forcing myself off HN lately. One thing that I miss the most due to this is the SHOW:HNs of really cool products and hacks that hit the front page. This gives me a great month-to-month access to those great stories at one place.

Bookmarked!

duck
I have been forcing myself off HN lately

You might want to checkout my newsletter which includes the best "show hn" links each week - http://hackernewsletter.com. It also helps with the HN addiction.

TeMPOraL
I've been subscribed to it for some time and while it didn't cure my HN addiction, it lowered it a bit, and it also helps me find interesting articles I missed during "more work, less procrastination" periods. I just want to say, thank you for your work!
j45
It would be useful to have a category for startups. I find that's the content I most look for here.
d_luaz
Tagging of links (mostly hn), without the NLP. http://gcdc2013-hackerio.appspot.com/
dclara
What's the difference from the online bookmarks such as del.icio.us?
d_luaz
1) Tech/HN centric links and tags/categories 2) Besides bookmarking and tagging, it also support list (a collaboratively editable list on topic such as SEO tips)

http://blog.gcdc2013-hackerio.appspot.com/what-is-hackerio

dclara
You are doing something very similar to what I've done for two years. But I'm running a serious business and startup company with patent pending solutions.

Another similar thing between us is that when we showed HN for our projects, we both got 0 comment from the HN community.

So the question is why they are not more welcomed than OP's NLP solution, even if we provide more tools to save and share?

Do you have any idea? If you are interested in further discussion, I can be reached at danmark.clara _at_ yahoo.com.

d_luaz
Because NLP is more interesting? We suck at community building? Not enough content? No HN Luck? Perhaps it's neither interesting or good enough yet :)
dclara
Now the new search engine algorithms are gearing towards NLP led by Google's Hummingbird, see the article here:

http://www.wired.com/insights/2014/02/search-today-beyond-op...

The direction is right because we all need to improve the web towards Semantic Web which is more meaningful. However, the first round of Semantic Web effort is dead: http://bit.ly/JP3KQO

Now it's the second round. Still the machine intelligence is not comparable with human intelligence. So what I believe is the way we are applying: organize the web with human interaction and automated it with some facilities.

I don't think we suck. It takes time for others to dive into this world. So I'll continue to make this effort and push forward. The problems for yours and mine are also pretty common: UI is not appealing, lacking of data.

I know you are running it for fun. But if there is any chance for us to work together, I think it's the best, because you understand this area and solution so well. Check out my failed Kickstarter project here http://kck.st/JNqv8z and give a try of my existing website: http://bingobo.com. I'll appreciate your feedback.

cogware
I find the font uncomfortably small - make the styling more like HN?
federkasten
Thanks. I will support responsive web design. It maybe solve it.
sharjeel
Could you please add a category "All but NSA"
collypops
Isn't that just news.ycombinator.com?
xerophtye
Nope, I see NSA news every other day on HN and it is getting rather annoying
Houshalter
Supposedly there is a filter already on HN for NSA articles, each vote only counts as 1/3 of a vote if the title contains the string "nsa", severely penalizing those articles.
tempodox
If that's true, I understand why I don't understand the HN sort order.
baddox
You could also call it "all but the single most important issue facing the technology and computing industries today," but your proposal is more concise.
[deleted]
(comment deleted)
ewoodrich
Very nice, would it be possible to have a larger selection of categories, and the option to exclude certain categories for a more fine grained front page?

Also, I believe you haven't escaped the time field properly server-side, I see "item.time_str" where I imagine the item time should be dynamically displayed.

minikomi
Just a thought - how about having the categories "toggle" rather than "select"?
herokusaki
Out of all third party UIs for Hacker News I've seen (and have since forgotten the URL for) you probably have the best domain name. It's clever and memorable.
11185d
This is great! I would also love the ability to collapse entire threads, this is something I never understood why it doesnt exist.
tomkinson
It is very useful, although I do love simplicity!
jasonkester
It's a sad commentary on the current demographics here that there is no "entrepreneur" category at all. And that the next closest thing, "business" has only 3 articles, none of which are related to running a business.

I'd noticed myself finding less and less to click on on the front page over the last year, but this has really driven home why.

nkuttler
Well, it seems you can't have so much success and preserve original community values. We've seen that over and over on various websites.

If you haven't seen it yet, try http://hackermonthly.com/. I'm not a subscriber, but the free issues I've seen had more business stories in them. You can download the current issue from http://hackermonthly.com/issue-45.html

swah
May I ask, how do you use HN these days?
ScottWhigham
I'm not sure that the "business" category means what you or I expected it to mean. It's certainly nothing to do with being an entrepreneur - as I write this there are four articles:

DEA redacts tactic that's more secret than parallel construction

Jamaican Bobsledders Ride Dogecoin Into Olympics

Swiss police: Screen in Tesla cars is too large

Ask HN: What will you pay for?

============

I guess maybe that last one could be somehow related but IDK. Seems that important things are missing from the categories (like entrepreneurship or running a business, which are not the same thing). In another comment on this thread, it was mentioned that these are auto-tagged - maybe they can train it a bit better. Those are, to me, off-topic for 2008 HN but right in the wheelhouse for 2014 HN.

davidw
> Those are, to me, off-topic for 2008 HN but right in the wheelhouse for 2014 HN.

The guidelines haven't actually changed any since then.

ScottWhigham
haha yes, but would they have made the front page in 2008? I think you know the answer to that one!
level09
looks like a lot of data have not been indexed yet, the concept is extremely useful, and would like to see psychology as one of the categories, they are my favourite type of content on HN.
ColinWright
It would be useful if each article could have its tag displayed. In that way when looked at "all" I could start to get an idea of what the tags mean to the system. At the moment I'm reasonably often finding things tagged other than I would expect. Showing tags next to items would help me recalibrate.

Other than that - good job. Thank you.

federkasten
Thanks.

You're right. That's nice idea! I will add that feature shortly.

tempodox
I never got the original HN sorting algorithm. Looks quite arbitrary. So, any alternative is likely to be an improvement.
dhimes
I like this alot. BUG: item.time_str isn't evaluating when I select a category.
acd
Brilliant work thanks!
federkasten
Updated.

1. fix some bugs (e.g. "item.time_str")

2. support responsive design (currently optimized with iPhone)

federkasten
Updated.

* fix some stylesheet issues (support Chrome on Android)