Seems like there's an API if you want to get some datas out of it. Otherwise, you might want to try to use javascript on trusted websites.. it actually often enhance the pages.
Awesome. One can also use it to analyse various trends on HN. For ex. look at this story: http://news.ycombinator.com/item?id=4739649 from November 4th 6:40 pm to 11:00 pm. Shows you how HN'ers are eager to help each other.
There was something very similar to this for reddit that was very handy and useful. It was at redditsnapshot.sweyla.com but it shut down at some point. I still think there is an opportunity for someone to index reddit's front page and have it hour by hour and maybe a premium version of all the subreddits you subscribe to.
When I saw that sweylas's stuff has disappeared, I asked them in an email if it will be back again. I didn't get an answer so I implemented my own version: http://redditsnapshots.com/ (running since March 2012)
Loved this. I would love the feature where I could "highlight" (add a glow for effect) any particular story and watch it move up and down as I play with the arrows. Right now everything moves, so this would help spot trend for a particular story.
Very nice. I actually thought of implementing this a few months ago but a quick Google search brought up http://hackerslide.com/ (also open source), which works fine as well.
I've learnt two things from this project in particular. First, that most similar projects don't seem to stick around very long (the Reddit one it was based on disappeared after a few months as have several others - http://hckrnews.com/ is an exception I can recall). Second, these tools seem to be popular at first but then rarely used over time. Luckily I still find it useful to catch up after vacations, etc ;-)
"Perhaps one minor bonus to HackerSlide for now is anyone can take the data collected."
Indeed, may I make a suggestion here:
1) Why not make a datadump so people wouldn't need to scrape ~800*24 json files individually?
2) OP ought to load this data into his version so the timeline goes back further
3) It seems quite a few people get the urge to tinker like this with HN, I'm sure pg doesn't mind the scraping, but it strikes me as vastly more efficient if some sort of shared resource was setup and perhaps added to the footer, in the vain of HNsearch, so people don't waste time get crawling data setup.
I'm sure somebody else has a dataset just like yours that goes back further still. :)
Also, thank you for making this and OP for making his. Fun.
The front-end specifically asks for pages in 200-chunks. So whether you slide to page 299,300 or 385, it will request 200-400 from the API. This means I can then very easily serve these requests out of Redis [1] and during usage spikes, requests never hit the disk :)
I see that you are using pyquery to scrape the content. I was using BeautifulSoup for my previous projects, but it seems pyquery is a better choice due to its compatibility with jQuery, so I am planning to switch too. Are there any downsides to pyquery, though?
This looks amazing! Good job! If I may ask(yes I have taken a look at your code but unfortunately my python is not too good), whats the "secret" to grabbing HN's historic data? Also, would it be possible to go back beyond one month towards maybe a year or even few years back (idk if it is just me but I can go up until October 9th, 2012)?
This is where the scraping happens. The code is a little uglier than I'd like, but that's largely to do with the hard-extract-data from HN markup. I'll look at adding more data soon.
53 comments
[ 15.7 ms ] story [ 1154 ms ] threadDoes the keymaster live demo at http://madrobby.github.com/keymaster/ work for you?
I originally just dumped the raw scraped HTML into the DOM but that proved to be too space-inefficient (even with gzip).
All in all, Awesome.
Perhaps one minor bonus to HackerSlide for now is anyone can take the data collected. The URLs for the JSON archives are formatted like: http://hackerslide.com/data/2012-11-01-23.json (YYYY-MM-DD-HH) although even better long term are the once a day versions, e.g.: http://hackerslide.com/data/2012-11-01.json
I've learnt two things from this project in particular. First, that most similar projects don't seem to stick around very long (the Reddit one it was based on disappeared after a few months as have several others - http://hckrnews.com/ is an exception I can recall). Second, these tools seem to be popular at first but then rarely used over time. Luckily I still find it useful to catch up after vacations, etc ;-)
Indeed, may I make a suggestion here:
1) Why not make a datadump so people wouldn't need to scrape ~800*24 json files individually?
2) OP ought to load this data into his version so the timeline goes back further
3) It seems quite a few people get the urge to tinker like this with HN, I'm sure pg doesn't mind the scraping, but it strikes me as vastly more efficient if some sort of shared resource was setup and perhaps added to the footer, in the vain of HNsearch, so people don't waste time get crawling data setup.
I'm sure somebody else has a dataset just like yours that goes back further still. :)
Also, thank you for making this and OP for making his. Fun.
Publicly available JSON files was just a side effect of the implementation. But it's easily to tar and gzip it up, so there's now such a file at http://secretshenanigans.s3.amazonaws.com/hnfrontpages.tar.g... (32MB).
I would be sad if it disappeared.
I would love to see a list of all the Hacker News "meta tools".
I posted my own last week: http://www.HnEasy.com sorts hn all posts and comments by upvotes from the last day... to last 5 years.
I created this for the HN community and I'm very happy to see so many people enjoying it :)
If you have any questions about how I built this or if you'd like to suggest something new, let me know!
The front-end specifically asks for pages in 200-chunks. So whether you slide to page 299,300 or 385, it will request 200-400 from the API. This means I can then very easily serve these requests out of Redis [1] and during usage spikes, requests never hit the disk :)
[0] https://github.com/doda/rewindhn/blob/master/static/rewind/j...
[1] https://github.com/doda/rewindhn/blob/master/server.py#L25
i7 with 16 gig RAM
http://worrydream.com/LadderOfAbstraction/
Thanks!
This is where the scraping happens. The code is a little uglier than I'd like, but that's largely to do with the hard-extract-data from HN markup. I'll look at adding more data soon.