As quite a few have asked via the comment form: it's a nginx + passenger + ruby 1.9.2 stack, with solr for indexing, mongodb for data back-end, rails fragment caching in redis, and a warmup script to fill the caches.
Well I'm more used to Redis, as I've used more than a couple of times with Resque (job processing in Ruby).
I started the crawler behind HackerBooks with Resque and Redis, but ended up saving some RAM by going back to a simple daemon. I reused Redis to provide the caching.
on that note,
hackertext, hackermenu, hackerledger, hackerconventions, hackerelite, hackerplans, hackermanifest, hackerstatement, hackerplay, hackersection, hackervolumes, etc... all seem to be free. (shameless plug for nametoolkit.com - downvote me if you wish ;))
Thank you for building this. I admire the simplicity and elegance. Would be cool if the "Mentioned 2 times at Hackernews" was hyperlinked to the mention. Can this be done?
On parsing: it's actually a somewhat fastidious process that involve digesting a couple of GB of data, but here is the bottom line.
I look for amazon.com links in the content in general - I will broaden that to other publishers and full-text extraction too later on.
The content itself comes from the StackOverflow dump (for SO) and a mixture of a crawler allowed by PG + the previous database dump that was available at some point.
I extract all the books, quotes, users data from both, conform these into a common schema, and index the whole result.
Hope I answered your question properly - feel free to ask again if you'd wish.
On ranking: I know what you mean! I need to find some way to balance number of quotes with textual relevance, which requires me to dive a bit more into solr. I currently use textual relevance first because it gives more useful results so far.
For ranking, instead of textual relevance (which will be hard to achieve :/) and # of quotes (it can be easily hacked/spammed or new announced books will have a huge weight on the ranking), I suggest you to check timelines: if a book is quoted once/twice/... a month regularly, I'm pretty sure it worths reading it
I'll love to read about the architecture behind the site... yep, technically curious :)
I will offer a more advanced search with similar features, definitely!
On the architecture: I'll create a side-blog that will outline all I learned while working on this. It's been a crazy ride actually (especially because I started using chef and vagrant full speed).
Questions:
Apart from the "Quoted by" section, is any of the content from SO or HN displayed on the site? E.g. are any comments incorporated in the book descriptions, or are those written by you/wife?
Currently, no content from HN/SO is displayed apart from the Quoted by area.
We're not editing anything manually; what I will do is display the actual conversations in the "Quoted by" area, either when you click on a conversation.
I may ove the quotes above to make them stand out more.
Aaah - I see :) The book description is provided by the Amazon API itself; I'm not aware of any issue with publishing it if you are a registered API user.
Tweaking relevance is not super-easy with Solr. We designed IndexTank to have a very simple way to play with ranking. You can modify your formulas in your dashboard or through the API and see the results order change in real time.
I really love what you got here. I'd be happy to help you try out IndexTank and make it better. It would really take the Solr configuration burden off of you.
well I considered using IndexTank earlier on, especially because I didn't know yet how to deploy Solr. The relevance is mostly done already, I just needed to learn to use formulas :)
One thing that put me off is your cap in queries per day. The smallest paid plan (50k items in index) is capped at 1,000 queries per day.
Isn't that an issue for most sites ? Do people usually cache your results ?
The 1k cap for the 50k doc plan is old, we will be upping that significantly. What would be a number of queries per day that would make you comfortable?
I'm a nobody, but consider making the pages less clicky and more product page-y, with the external references that provide the structure of your site constituting something of a "topics covered in this book." I think the references/links should be at least as prominent as the book blurb that currently serves to authoritatively describe the contents. My point is that the contents may actually be described (for sufficient numbers of refs) better by listing how people are actually recommending it, how its readers are using it. You could even implement a semantic parser that constructs a new book blurb based on the words and sentences in the recommendations and references. :) Anyway, good luck!
I dont know if there is an API for it or not but if you can get the Amazon rating of a book and display it on the book description page, it will be great. :)
On AZ ratings: I wrote that down. It's somewhat complicated because Amazon just made it a bit harder to embed that. It now has to be an iframe; the iframe url must be refreshed every 24 hours.
But overall this should be doable to, I'll see if I can make it usable.
That was what I was doing initially, but actually it has (big) sideeffects: if you search Ruby and get the most quoted book, you get... an ASP.Net book!
So I'm currently working on balancing textual relevance (eg: Ruby) with the number of quotes.
Useful, and also a neat way to make some side-cash via your affiliate link. I suspect I'll be checking this regularly.
One thing that would be really handy - or at least interesting - is a "most recently mentioned" list. For example, when people were talking a lot about Program or be Programmed a while back, it would have been fun to see that rise to the top.
It may generate some side-cash via the affiliate links - it would certainly be useful to us. We'll see how it goes anyway :)
The most recently mentioned list is a very good idea. I had something similar in mind, like a news-letter that would send the "most mentioned this month", so you can get the trends.
I think it could be useful for sure. Both a most recently mentioned list, and most often mentioned list. I'm more interested in the latter as it is more telling.
If making cash via an affiliate link is the goal, it would make sense to have the link to the user's localised Amazon site. I much prefer to buy from Amazon.co.uk as I'm UK-based so right now I'll just highlight-> right-click-> google the title and end up bypassing the affiliate link.
Well it's only a secondary goal, as I don't expect to make loads of money with it, but it would certainly be nice.
Being able to select your amazon store is planned as well (I'm in France so I totally understand your point :-)).
I choosed to ship without that though, to see if people like the site first or not. It will require a bit of work underneath to do well, things such as verify in the background if a book is actually available on amazon.co.uk, .fr etc to avoid sending the person to the wrong place.
This site is really a good idea. The Amazon links can get quite popular (I know from looking at my Amazon stats on @hackerlinks when I had the affiliate code inserted.)
Wow, good idea. I've browsed it this morning and it's an excellent website. Some things I might add is a link to the context of where the book was cited and also better categorization. (i.e. effective C++ in compiler book). But otherwise, amazing. Thank you
It's probably not too difficult, although my guess is you'll have to refeed your content. Unfortunately I've never worked with solr so I can't offer suggestions on where the modifications need to be made.
Awesome. I recently ordered the well-grounded rubyist as I've heard great things about the book (though its on page #2 on your site). How long did it take you to complete the project?
Since I started freelancing in 2005 I've always deployed my stacks myself, learning gradually.
I always felt that doing it manually (even using well-written notes) then gradually home-baked tools was a loss of time.
I looked at chef more than a couple of times, waiting for the documentation to be more available, and for feedback from people I know.
I started using chef with the opscode platform, then went back to chef-solo as it really fits my needs already.
I'm using it for client work as well as for everything behind HackerBooks (including Rails app deployment without capistrano anymore).
The consequence is that I can boot a new ubuntu instance from scratch, completely configured with the whole stack (rvm, rails 3, passenger, nginx, solr, god, the properly configured crawler, data restored from a S3-like etc) in less than 15 minutes.
I will never go back to manual sysadmin (apart from small tips) - this really fits my way of working.
But it has been a time-sink to get in :)
Hope I replied to your question properly, if I didn't, ask again!
I wonder sometimes if it'd be worth creating a site that aggregates all the most useful and interesting information from social news sites like Reddit, HN, etc. like this site does for books on HN. I spent a long time finding all the best "life hacks" on Reddit the other day and really found some gems.
I really liked the site, congratulations. Just because I am a stack freak, can you tell us more about your technology stack and how fun (or not) was to develop this?
Yes I know about Daniel's site, which was released while I was working on hackerbooks.com. I decided to keep working on it anyway, but it took me a fair amount of time to release.
I recently registered fivegoodbooks.com with the plans of trying to reduce choice aversion for technical books. i.e. the top 5 referenced books by the community to start learning a subject. This is really nice work, now I'm the one to have to decide if I keep working on it :)
Thanks for your kind words! Well I can see myself using both sites, really :)
That' what I learned from working on http://www.learnivore.com too - at the same time came out http://rubytu.be/, but I used both actually, and some people preferred one, some other the other.
Nice domain name... got me thinking. I wish there existed a community I respected that curated books on all topics. Like, I'd love to have fivegoodbooks extend from topics like Ruby and Python to Russian History and Urban Planning.
Another interesting idea maybe would be to map the annotations/bibliographies of all books, so I could start with books I've read and better see what kind of linkings exist. You could visually see the seminal works in a field and all the branchings. Maybe that already exists in some form?
Interesting. I had a shower idea(tm) one morning where the combination of a rating and user-supplied biographical data at the time of the rating (e.g. "I was a novice with Python when I read it") determined a book's effectiveness.
The idea was founded on the tendency that people are skill-biased when rating books. They might dislike a book for being confusing or too easy because it wasn't designed for them at the time of reading. This was an attempt to figure out which books were just bad, and which were only rated bad because they were read without proper experience (or too much).
Ultimately, using community data, a visitor would be able to discover a "bookpath" of great resources that syncs up with their level of skill.
The wiki approach would be great if the data was laid out appropriately. Community-edited wiki might lead to a singular view of what a good path would be.
The benefit of a multi-axis rating system is that you could lay out data using different permutations and add/subtract inputs (e.g. rater's experience at time of rating/age/hell, maybe even personality type). I'm sort of modeling this off of robust scientific questionnaires.
It does seem to require some more user involvement, but nothing a fantastic UI couldn't fix.
I really like this idea, especially if the "bookpaths" converge.. i.e. the ruby path and python path converge to broader topics such as general software development (pragmatic programmer etc.)
Site looks great - really great - on a macro level, but when you get down to it, it doesn't really have the content I'm looking for.
There's extreme misuse of the space on the page. When I was looking at Code Complete (a book I've been trying to get my hands on for a while), there is very little content about the book. The synopsis is cut off (!) and there are no reviews. But if you were trying to save space, why on the hells are there over 9000 books following in "quoted discussions"? You need to switch what you're truncating here. Also, I would suggest at least copying Amazon's ratings for some measure of book quality.
Hey - I understand what you mean, really. I really want to work on this specific page, and I'm actually frustrated to have released it this way :)
Here's what's planned:
- instead of cutting the book description like it's done currently, you'll be able to expand with a click
- I'll do the same on books and quotes, because hundreds of books are not useful in the "suggested books list", definitely
- and I want to focus more on getting the actual quotes to the front (eg: allow to read the actual quotes by HNers etc) because I feel it brings significant usefulness
I'm mixed on the Amazon ratings, both because recent API changes made it unpractical to really use the content (it's now an iframe that must be used as is, and refreshed every 24 hours), and because sometimes the reviews are fake as well.
So I'll try to bring more value by letting users know what people think on HN and SO.
Would these points make the site more useful to you ?
On the synopsis: well in some (too frequent) cases, the synopsis was just several pages long, so the related books and quotes were really, really hidden.
I need to find a better middle ground.
But in all cases, thanks for your critics, it will only help me make the site better.
143 comments
[ 2.2 ms ] story [ 216 ms ] threadIt's fairly simple so far and more features are planned.
I'm submitting early on to get some wider feedback. Thanks to all the HNers that reviewed this before today already!
I started the crawler behind HackerBooks with Resque and Redis, but ended up saving some RAM by going back to a simple daemon. I reused Redis to provide the caching.
So well: I mostly used what I was used too.
In the end I paid 90$ for 3 years, which is OK I guess.
Or was it the opposite?
Wish us luck :) (and communication, it's needed :-)
Next iteration will be on making the navigation better, so that you can click on everything that's possible.
Keep the good work. I love it :)
My only suggestion is to offer a ranking and order the results by # of quotes
On parsing: it's actually a somewhat fastidious process that involve digesting a couple of GB of data, but here is the bottom line.
I look for amazon.com links in the content in general - I will broaden that to other publishers and full-text extraction too later on.
The content itself comes from the StackOverflow dump (for SO) and a mixture of a crawler allowed by PG + the previous database dump that was available at some point.
I extract all the books, quotes, users data from both, conform these into a common schema, and index the whole result.
Hope I answered your question properly - feel free to ask again if you'd wish.
On ranking: I know what you mean! I need to find some way to balance number of quotes with textual relevance, which requires me to dive a bit more into solr. I currently use textual relevance first because it gives more useful results so far.
We'll see how it goes.
For ranking, instead of textual relevance (which will be hard to achieve :/) and # of quotes (it can be easily hacked/spammed or new announced books will have a huge weight on the ranking), I suggest you to check timelines: if a book is quoted once/twice/... a month regularly, I'm pretty sure it worths reading it
I'll love to read about the architecture behind the site... yep, technically curious :)
On the architecture: I'll create a side-blog that will outline all I learned while working on this. It's been a crazy ride actually (especially because I started using chef and vagrant full speed).
I'll post it back here in all cases.
Questions: Apart from the "Quoted by" section, is any of the content from SO or HN displayed on the site? E.g. are any comments incorporated in the book descriptions, or are those written by you/wife?
Currently, no content from HN/SO is displayed apart from the Quoted by area.
We're not editing anything manually; what I will do is display the actual conversations in the "Quoted by" area, either when you click on a conversation.
I may ove the quotes above to make them stand out more.
Did I properly answer your question ?
Was justing wondering if the content might be touched by any copyright issues etc.
I really love what you got here. I'd be happy to help you try out IndexTank and make it better. It would really take the Solr configuration burden off of you.
well I considered using IndexTank earlier on, especially because I didn't know yet how to deploy Solr. The relevance is mostly done already, I just needed to learn to use formulas :)
One thing that put me off is your cap in queries per day. The smallest paid plan (50k items in index) is capped at 1,000 queries per day.
Isn't that an issue for most sites ? Do people usually cache your results ?
I can't really tell yet, but my guess is that at least around the number of indexed documents could give a more usable subscription.
It would also be nice for people to know what happens if you go beyond the cap: do you offer some tolerance ?
I dont know if there is an API for it or not but if you can get the Amazon rating of a book and display it on the book description page, it will be great. :)
On AZ ratings: I wrote that down. It's somewhat complicated because Amazon just made it a bit harder to embed that. It now has to be an iframe; the iframe url must be refreshed every 24 hours.
But overall this should be doable to, I'll see if I can make it usable.
Thank you!
edit: not a mix, but separate points for each community
I think being able to filter on SO vs HN would definitely give different results, too.
The topics are sometimes fairly different on both sites.
So I'm currently working on balancing textual relevance (eg: Ruby) with the number of quotes.
Thanks for your feedback!
One thing that would be really handy - or at least interesting - is a "most recently mentioned" list. For example, when people were talking a lot about Program or be Programmed a while back, it would have been fun to see that rise to the top.
The most recently mentioned list is a very good idea. I had something similar in mind, like a news-letter that would send the "most mentioned this month", so you can get the trends.
Would you find that useful ?
My pet peeve is allowing to find books quoted by X, where X is some instance of someone I appreciated on HN :)
I think I will create a labs section with various experimentations like these, so people can try them out and see if it's useful.
Being able to select your amazon store is planned as well (I'm in France so I totally understand your point :-)).
I choosed to ship without that though, to see if people like the site first or not. It will require a bit of work underneath to do well, things such as verify in the background if a book is actually available on amazon.co.uk, .fr etc to avoid sending the person to the wrong place.
In this respect you'll see new Amazon links every so often in: http://twitter.com/hackerlinks (the tweet will begin with Amazon) http://hackerbra.in/links
This site is really a good idea. The Amazon links can get quite popular (I know from looking at my Amazon stats on @hackerlinks when I had the affiliate code inserted.)
I didn't know about hackerlinks either - just subscribed!
Good luck!
Perhaps I should make an Amazon only Hackerlinks-style feed! Or you could have one coming from your site of new books as they're added!
I plan to display the actual conversation in some way on the site itself, if I can. Lot of people told me it would be useful.
I will also work on some topic extraction, yes!
I wish I'd thought of it.
I know for sure it can be done with some configuration tweaking, not even sure I'll need to reindex.
Thanks!
Thank you for mentioning this!
I searched PHP and would like to see by # of recommendations...
It's currently sorted (in that order) by 1) textual relevance and second 2) number of quotes.
I plan to write a couple of blog posts explaining the "how" later on! It's been an interesting ride really.
But a large part of it (the 2/3rds) was a learning exercise around chef and vagrant, which wasn't necessary to the project.
Of the remaining third, I've got around 70% for data processing in general and 30% on pure front-end code and design.
I really wanted to learn how to deal with sysadmin in a more productive fashion, so I took the plunge :)
I always felt that doing it manually (even using well-written notes) then gradually home-baked tools was a loss of time.
I looked at chef more than a couple of times, waiting for the documentation to be more available, and for feedback from people I know.
I started using chef with the opscode platform, then went back to chef-solo as it really fits my needs already.
I'm using it for client work as well as for everything behind HackerBooks (including Rails app deployment without capistrano anymore).
The consequence is that I can boot a new ubuntu instance from scratch, completely configured with the whole stack (rvm, rails 3, passenger, nginx, solr, god, the properly configured crawler, data restored from a S3-like etc) in less than 15 minutes.
I will never go back to manual sysadmin (apart from small tips) - this really fits my way of working.
But it has been a time-sink to get in :)
Hope I replied to your question properly, if I didn't, ask again!
==== ( http://asso.in/url/hp ) ====
Christan Audigier bikini $23
Ed Hardy Bikini $23
Smful short_t-shirt_woman $15
ed hardy short_tank_woman $16
Sandal $32
christian louboutin $80
Sunglass $15
COACH_Necklace $27
handbag $33
AF tank woman $17
puma slipper woman $30
==== ( http://asso.in/url/hp ) ====i ~ ¤ ╭⌒╮ ╭⌒╮ ╭⌒╭⌒╮╭⌒╮～╭⌒╮ ,））））,''）~~ ,''~） ╱◥█◣ ╱◥█◣ ｜田｜田｜｜田｜田｜ ╬╬╬╬╬╬╬╬╬╬╬╬╬╬
I will probably come up with some kinds of list, and will definitely work on better sorting/filtering.
That' what I learned from working on http://www.learnivore.com too - at the same time came out http://rubytu.be/, but I used both actually, and some people preferred one, some other the other.
I encourage you to ship anyway :)
Another interesting idea maybe would be to map the annotations/bibliographies of all books, so I could start with books I've read and better see what kind of linkings exist. You could visually see the seminal works in a field and all the branchings. Maybe that already exists in some form?
The idea was founded on the tendency that people are skill-biased when rating books. They might dislike a book for being confusing or too easy because it wasn't designed for them at the time of reading. This was an attempt to figure out which books were just bad, and which were only rated bad because they were read without proper experience (or too much).
Ultimately, using community data, a visitor would be able to discover a "bookpath" of great resources that syncs up with their level of skill.
I wanted to create a community-edited wiki page providing a "training path" which would involve screencasts, books, articles.
I think it would really be useful actually.
The wiki approach would be great if the data was laid out appropriately. Community-edited wiki might lead to a singular view of what a good path would be.
The benefit of a multi-axis rating system is that you could lay out data using different permutations and add/subtract inputs (e.g. rater's experience at time of rating/age/hell, maybe even personality type). I'm sort of modeling this off of robust scientific questionnaires.
It does seem to require some more user involvement, but nothing a fantastic UI couldn't fix.
I would love to be able to build something like you describe, but I suspect it will be time consuming, that said!
There's extreme misuse of the space on the page. When I was looking at Code Complete (a book I've been trying to get my hands on for a while), there is very little content about the book. The synopsis is cut off (!) and there are no reviews. But if you were trying to save space, why on the hells are there over 9000 books following in "quoted discussions"? You need to switch what you're truncating here. Also, I would suggest at least copying Amazon's ratings for some measure of book quality.
For reference: http://www.hackerbooks.com/book/code-complete-a-practical-ha...
Here's what's planned:
- instead of cutting the book description like it's done currently, you'll be able to expand with a click
- I'll do the same on books and quotes, because hundreds of books are not useful in the "suggested books list", definitely
- and I want to focus more on getting the actual quotes to the front (eg: allow to read the actual quotes by HNers etc) because I feel it brings significant usefulness
I'm mixed on the Amazon ratings, both because recent API changes made it unpractical to really use the content (it's now an iframe that must be used as is, and refreshed every 24 hours), and because sometimes the reviews are fake as well.
So I'll try to bring more value by letting users know what people think on HN and SO.
Would these points make the site more useful to you ?
On the synopsis: well in some (too frequent) cases, the synopsis was just several pages long, so the related books and quotes were really, really hidden.
I need to find a better middle ground.
But in all cases, thanks for your critics, it will only help me make the site better.