I built this small app in my spare time to aggregate books recommended on Hacker News. I personally find books recommended on HN to be super helpful, so I think this is the way that I can contribute back.
This book aggregation idea is not new. A bunch of sites have done similar things [1, 2, 3].
Yet one common limitation of those sites is that they have limited recall (i.e. not able to get a comprehensive set of book mentions), and thus don't paint an accurate picture of what the top books are. They're all based on insufficient rules, e.g., looking for Amazon Links. As you can see from my app, people often do not include Amazon links when recommending a book.
I wondered, why can't we just match book names? Well, not so easy. Some books have pretty short names, e.g. Meditations [4], or Steve Jobs [5]. Some book name might as well be the name of a movie, e.g. Ready Player One [6]. Simply matching the names of the books would produce a whole lot of irrelevant results.
This is where Deep Learning comes into play. Recent advances in large NLP models (transformers and BERT in particular) have made machine language understanding unprecedentedly accurate. It enables me to fine-tune a BERT model on a couple thousand labeled HN comments and predict accurately whether each word in a comment is part of a book or not - a task commonly termed as Named Entity Recognition (NER).
As a result, my app is able to present a whole lot more results while maintaining desirable accuracy. For example, NER works pretty well on the tough examples I mentioned ([4, 5, 6]). Compared to prior sites, my app captures 9-50X more mentions and thus presents a much more complete picture of what books are recommended on HN.
Furthermore, I've made sure that the comments are presented well in the UI because the recommendations are just as useful as the books. I highlighted the mentioned book name, and used a custom NLP-based ranking function to sort the comments. These are non-trivial improvements over prior sites, which I hope you can find useful.
Nevertheless, this app is not without limitations: 1) matching book names would fail when two books have the same or similar names; 2) although not often, this approach would wrongly classify some short stop-word names [7] and 3) sometimes NER fails to see that the commenter actually hates the book. These problems can be alleviated with more Deep Learning. For 1), one can use BERT to learn the authors mentioned which can be used as a filtering criteria. 2) and 3) should be fixable with more training data (currently there are only ~4,000 hand-labeled HN comments).
Lastly, I'd like to especially thank my gf who helped me label ~1,000 comments, which boosted the model accuracy by 5 percent! I also want to thank the people who create and maintain the HackerNews big query dataset [8]. And of course, thank everyone on HN who recommends books to others.
Hope you enjoy this app! Feedback and suggestions are welcome :)
Great work but do note that the list basically looks slightly better than an amazon list (atlas shrugged lol). I think some effort into more useful ranking (looking for metrics of controversiality or maybe page rank) might make it more useful!
I am also curious to know if the # of votes is integrated into the ranking at all, possibly weighted. Could also attempt NLP Text Sentiment analysis to influence the model as well.
This is awesome. The best thing is that it's so fast to navigate. I like how the HN comments are styled just like on HN.
A couple of thoughts:
* It would be great if each book were to have its own URL (for sharing).
* Consider allowing the search to allow author input, e.g. if I want to find the book 'Who' by Geoff Smart, the single-word title isn't specific enough to show that book at the top of the search results.
If the dataset were perfect, maybe. But, if a book with a single-word title has only few comments, it's plausible that most/all of those comments are false matches.
In the case of the book I searched 'Who', showing it in 4th position seemed about right.
HN really likes Neal Stephenson. I've never read a book of his that I didn't love, so will be definitely looking though more of the recommended fiction from the community here.
I really liked Seveneves, but totally understand why someone wouldn't like parts of it. I am curious as to which third you skimmed though as the book is effectively three different books with very different moods.
The first third is very intense, I also was a bit fatigued for that middle third. When I have reread Seveneves I have skipped it. I really like the last third, but it does feel like a completely different book.
Only one I haven't read. Skip it I guess. Reading DoDo now, so verdict's out, but I absolutely adored every other of his books. Fall as a pleasant surprise, since it seemed controversial.
Reamde would have been my first recomendation to introduce someone to Stevenson. Constant action, an unusually direct storyline for him, and not too too much time spent off in the weeds.
Of course some might argue those are the best things about his books, but while Chryptonomicon is perhaps my favorite sci-fi novel I've read, I think it takes a certaim type of person to enjoy a book where the plot gets interupted for 5 page descriptions on how to eat Captain Crunch or a who fucked who of the Greek pantheon.
You mentioned transformers and BERT for large NLP models. I've been playing around with this too and it's a really powerful approach. Have you used spacy-transformers? [0]
The approach is pretty cool and can be used with BERT, GPT-2/Hugging Face etc.
I'm just starting to experiment with GPT-J and thinking of trying this approach also [1].
Anyway, totally awesome project and the results are really good. This stuff really is almost unreasonably effective!
This is a really good application of it. Getting NER right for something like book titles with so much name collision with other domains and entity types is really hard, and this works great on something that most people would never realize would be so hard!
This is really impressive! Can you please elaborate more on the way you labeled the data? I think usually there is a lot to learn from labeling methods.
I generated training comments by matching book names. Roughly, there are one in five of those comments that actually have a book mention. Then I use the Doccano labeling tool to label the tokens in the comments.
Ah right now ranking is really rudimentary: I just use TextBlob's SA library to get a sentiment score for the comment, and combine it with the comment length. In theory we can also use BERT to get a sentiment score, which should be more accurate I guess? Love to hear your thoughts.
Simple is good! Especially for ranking as the objective is hard to define. Having looked at some samples on your site, imo it's good enough :)
Though if you wanted to try other things just for fun, maybe:
- Count matching NER, comments with a lot of book recommendations tend not to detail why they like the specific one currently filtered
- maybe down weight comments that are too short (after the matched NER is subtracted) as they seem to just have a title+link?
Not ML but while I'm here, on Android the comment section appear fine but has an horizontal scroll that seems spurious, with lots of blank space to the right (Galaxy S10, chrome & firefox)
This is great. I just read Permutation City, which I coincidentally see recommended on HN all the time, so I was surprised not to see it in the search results or the top of the fiction or scifi lists. Any idea why that is?
A few years ago I found an article that was something like '100 short books everyone should read before they're 40'. It was a mix of fiction and non-fiction. I've never been able to find it again! But I really liked the list because these are books you can consume in a few hours and may be life changing.
I remember a few of the titles: Games People Play, Meditations, The Prince, The Art of War. (I suppose it may have been non-fiction only, although I think The Awakening may have been on there.)
That's a bit like saying watching porn is more for personal development than fun. Perhaps you'll learn something, but it's incidental.
I've learned a lot from HN. But it wouldn't be good to fool myself into thinking that an employer wants to fund my personal development in this regard. Otherwise, they'd pay me to HN all day.
The crux of the issue is that it's impossible to work 8 hours every day. We all invent lies to fill the downtime.
Is all that hyperbole really necessary? Each new sentence seems primed to leak edge and corner cases. Without giving more attention to such a rhetorical blind spot, I wonder how one could imagine they know the crux from the passenger side door.
If it’s mistaken, it should be easy to explain why. Otherwise I’m inclined to believe it’s merely an uncomfortable truth.
Would your employer pay you to HN all day? If not, precisely how much of your day are they comfortable with you HN’ing? Are you sure it’s officially approved?
There’s often good intel on here that I have not been exposed to on other sources. Obviously spending 40 hours a week reading HN and getting into politics arguments on here is a waste, but there’s plenty of relevant news for most folks in tech if you stick to those topics.
Do you collect stats, how assertions line up with facts? Otherwise, what may seem likely and catchy, might just as well be opinionated, unsubstantiated and patchy.
The problem with reading book lists like this is that nobody has time to read all the books. That's a ton of crap out there and I want HN to help me filter through them.
Thus the problem with existing solutions is NOT "limited recall" or "insufficient rules" or "no Amazon link".
And the problem with this "solution" is that there is no justification for why a book is great and applicable to my circumstances, and people have to trust your black box. Otherwise I'm likely to waste my time, just like reading books from any other crappy recommendation engine.
With a deep learning model reducing all the reviews to "book names" you've successfully removed the value of the book discussions themselves. Therefore, for me this engine and all similar engines are strictly worse than simply going through the actual big threads themselves, i.e. https://news.ycombinator.com/item?id=21900498
Edit: I've just seen the embedded comments by switching to a desktop browser. It's a nice addition. However, for me to make sure I'm not wasting my time going through arbitrary books and comments, I would need to know why a book is ranked highly compared to other books. And I want to be sure that ranking is tailored to me, at a very, very high accuracy.
> With a deep learning model reducing all the reviews to "book names" you've successfully removed the value of the book discussions themselves.
It literally shows each comment in full that it extracted the book name from. It also includes a link to the comment in the original thread. What more could you possibly want?
Oh, I was on mobile and could not see the comments section. It's interesting for sure. But what I want in particular is to learn why a book is ranked highly compared to other books. And I want to be sure that ranking is tailored to me, at a very, very high accuracy.
There's no way around the black box element of a book review, but Nassim Taleb suggests waiting a few decades and, if the book is still well known, then reading it.
Wow. What a helpful piece of advice (I guess he's smarter than the rest of us so it's hard to understand the genius of his strategy). Any mention of the cost of missing out on the content in the book for a few decades?
Love this question. I could imagine him suggesting reading academic papers for cutting edge things; like his 'barbell' excercise strategy of mostly walking with occasional HIITs.
The idea behind reading older books is that they're already proven to be useful - it's a filter for you to spend less time on useless information. It's generally called the "Lindy effect"
What data source are you using for the books, authors and covers? I looked at OpenLibrary [1] but the covers are not the same, so I suppose it is something else? Maybe Amazon directly somehow?
Amazing and super useful: If I start reading today, and I read a book a day, it'll only take 112 years to finish, assuming that no additional books will be recommended in the next century.
I'm pretty sure there's more Warhammer 40k books than there are days in the year... It's like someone heard the term "space opera" and thought that meant "soap opera in space".
Recommendations would include comments like, "This novel is really the one that ties the previous 37 books together." or "You might want to skip the next dozen books if you're squeamish about things that ooze."
While I don't consider the 40k books on par with the better science fiction out there, I do enjoy that they bring a bit of scale and what it means to space. It's a different take from the rosy, post-scarcity, future of space. Bad things are really bad. Unattended good things turn bad on their own just from drift.
Then there is there is the unashamed embrace of over-the-top in so many different ways.
I've always considered 40k a satire of the entire sci-fi genre (and in many ways a satire on modern politics). In that way, I find it quite refreshing. And your statement of "unashamed embrace of over-the-top" resonates quite well.
Noticed an issue. Some, but not all, comments referencing Strunk and White's Elements of Style are showing up instead as Erin Gates' Elements of Style: Designing a Home & a Life
Good catch! This is the limitation mentioned in my root comment - the algorithm will fail when two books have similar names. The partial solution is to look at authors too when available. Something to be included in the future.
Interesting idea, but this is mentions of books, not recommendations. It includes comments by someone that's reading the book, has it on their reading list, or read it and thought it was terrible.
The intention was to only show recommendations. But because of limited training data (we hand labeled ~4000 comments), the model wasn't able to filter out bad ones effectively. More training data should be able to solve it.
"Atlas Shrugged" is a polarizing book: people tend to either love it or hate it. And the people who love it love to tell other people how great it is, whereas many of the people who hate it just don't talk about it (because there's generally little need to talk about the badness of bad books).
I think a book list is more useful if it has some books in it that some love and some hate, rather than only books that no one minds very much. Maybe some of them will turn out to be ones I love.
Slightly off from post. The best book recommendation I got is from one of the following ways, dedicated recommendation service or app never worked for me:
- told by friend
- someone I admire read book and commented on it
- I'm working on some problem, during that exploration i came across books.
- random people mentioning book on platform like HN on a topic/post of my interest.
> random people mentioning book on platform like HN on a topic/post of my interest.
Same! Some of my favorite book recommendations have especially come from this one, I don't know why but a one line comment on a HN thread of "what book changed your life" has become my favorite way for discovering books.
Yeah, in my experience a lot of 'general' book reviews are super critical and don't really try to hook you. Going through several reviews, you just come away with the collected gripes and nitpicks of what is otherwise a good book.
I find that I get sold on a lot more when it is just a random single comment on some thread somewhere that focuses on a single aspect of a book.
If you can find a hyper specific subreddit/forum/etc. for a sub-genre you like, then you will spend more time reading books than reviews...
I know this but I wasn't able to fix it. Would love suggestions on how to keep the scroll position in one div (for the books) but not the other (for the comments) when doing client-side navigation using Next.js...
I was going to make this suggestion too, but now that I saw this comment, instead I'll suggest asking on the next.js subreddit? It seems somewhat active, maybe someone there can help - https://old.reddit.com/r/nextjs/
It's a really interesting project. And I am sure it's really hard.
I was curious how many times some common textbooks were mentioned but didn't find them via the search, which could be user error. But to give a specific example. None of the books in this comment thread were found:
Does it take into account negative reviews/comments? I have seen that Why we sleep is being recommended in the 6 months tab, but, while it was received with a lot of praise, it was soon critizised by others researchers in the field and I would expect that the HN crowd would have followed that trend.
When I labeled the comments, I didn't label books that were criticized. So in theory the model should filter out negative reviews. But currently the training dataset is pretty limited in size so you still can see some negative ones. I suspect that with more training data this problem will go away.
284 comments
[ 3.1 ms ] story [ 304 ms ] threadI built this small app in my spare time to aggregate books recommended on Hacker News. I personally find books recommended on HN to be super helpful, so I think this is the way that I can contribute back.
This book aggregation idea is not new. A bunch of sites have done similar things [1, 2, 3].
Yet one common limitation of those sites is that they have limited recall (i.e. not able to get a comprehensive set of book mentions), and thus don't paint an accurate picture of what the top books are. They're all based on insufficient rules, e.g., looking for Amazon Links. As you can see from my app, people often do not include Amazon links when recommending a book.
I wondered, why can't we just match book names? Well, not so easy. Some books have pretty short names, e.g. Meditations [4], or Steve Jobs [5]. Some book name might as well be the name of a movie, e.g. Ready Player One [6]. Simply matching the names of the books would produce a whole lot of irrelevant results.
This is where Deep Learning comes into play. Recent advances in large NLP models (transformers and BERT in particular) have made machine language understanding unprecedentedly accurate. It enables me to fine-tune a BERT model on a couple thousand labeled HN comments and predict accurately whether each word in a comment is part of a book or not - a task commonly termed as Named Entity Recognition (NER).
As a result, my app is able to present a whole lot more results while maintaining desirable accuracy. For example, NER works pretty well on the tough examples I mentioned ([4, 5, 6]). Compared to prior sites, my app captures 9-50X more mentions and thus presents a much more complete picture of what books are recommended on HN.
Furthermore, I've made sure that the comments are presented well in the UI because the recommendations are just as useful as the books. I highlighted the mentioned book name, and used a custom NLP-based ranking function to sort the comments. These are non-trivial improvements over prior sites, which I hope you can find useful.
Nevertheless, this app is not without limitations: 1) matching book names would fail when two books have the same or similar names; 2) although not often, this approach would wrongly classify some short stop-word names [7] and 3) sometimes NER fails to see that the commenter actually hates the book. These problems can be alleviated with more Deep Learning. For 1), one can use BERT to learn the authors mentioned which can be used as a filtering criteria. 2) and 3) should be fixable with more training data (currently there are only ~4,000 hand-labeled HN comments).
Lastly, I'd like to especially thank my gf who helped me label ~1,000 comments, which boosted the model accuracy by 5 percent! I also want to thank the people who create and maintain the HackerNews big query dataset [8]. And of course, thank everyone on HN who recommends books to others.
Hope you enjoy this app! Feedback and suggestions are welcome :)
[1] https://news.ycombinator.com/item?id=15169611
[2] https://news.ycombinator.com/item?id=10924741
[3] https://news.ycombinator.com/item?id=12365693
[4] https://hacker-recommended-books.vercel.app/category/0/all-t...
[5] https://hacker-recommended-books.vercel.app/category/1/all-t...
[6] tinmandespot ↗ This makes me happy ramraj07 ↗ Great work but do note that the list basically looks slightly better than an amazon list (atlas shrugged lol). I think some effort into more useful ranking (looking for metrics of controversiality or maybe page rank) might make it more useful! vavooom ↗ I am also curious to know if the # of votes is integrated into the ranking at all, possibly weighted. Could also attempt NLP Text Sentiment analysis to influence the model as well. tracyhenry ↗ Right now the ranking is a simple combination of sentiment and length. Including #vote definitely sounds useful! rahimnathwani ↗ This is awesome. The best thing is that it's so fast to navigate. I like how the HN comments are styled just like on HN. soco ↗ If I look for one single word and that single word is the answer, shouldn't that be the very first result? I mean that's a 100% match right there... rahimnathwani ↗ If the dataset were perfect, maybe. But, if a book with a single-word title has only few comments, it's plausible that most/all of those comments are false matches. tracyhenry ↗ y the search can definitely be improved (e.g. to include author). Right now it's SELECT * FROM books WHERE name LIKE '%{search_string}%' MarcScott ↗ HN really likes Neal Stephenson. I've never read a book of his that I didn't love, so will be definitely looking though more of the recommended fiction from the community here. samuel ↗ REAMDE was crap, IMO, and I'm a Stephenson fan. macintux ↗ I addressed that problem with Seveneves by skimming about 1/3rd of it. ImaCake ↗ I really liked Seveneves, but totally understand why someone wouldn't like parts of it. I am curious as to which third you skimmed though as the book is effectively three different books with very different moods. macintux ↗ The middle third, since by the time I reached that point I was exhausted and ready to find out how it all ended. ImaCake ↗ The first third is very intense, I also was a bit fatigued for that middle third. When I have reread Seveneves I have skipped it. I really like the last third, but it does feel like a completely different book. the__alchemist ↗ Only one I haven't read. Skip it I guess. Reading DoDo now, so verdict's out, but I absolutely adored every other of his books. Fall as a pleasant surprise, since it seemed controversial. wrycoder ↗ Well, I liked it. The merging of different groups of people from different cultures and classes was zany, and the action was non-stop. gibspaulding ↗ Reamde would have been my first recomendation to introduce someone to Stevenson. Constant action, an unusually direct storyline for him, and not too too much time spent off in the weeds. jedwhite ↗ Hey this is really awesome! Well done. tracyhenry ↗ Thanks! I used Huggingface's pretrained BERT. jedwhite ↗ This is a really good application of it. Getting NER right for something like book titles with so much name collision with other domains and entity types is really hard, and this works great on something that most people would never realize would be so hard! sillysaurusx ↗ Please write up how you did this! It may seem easy or straightforward, but I assure you it's black magic to a lot of people. stanislavb ↗ That'd be great. tracyhenry ↗ You can refer to the medium post in this comment, which does more or less what I did: https://news.ycombinator.com/item?id=28601741 malshe ↗ This is really impressive! Can you please elaborate more on the way you labeled the data? I think usually there is a lot to learn from labeling methods. tracyhenry ↗ I generated training comments by matching book names. Roughly, there are one in five of those comments that actually have a book mention. Then I use the Doccano labeling tool to label the tokens in the comments. malshe ↗ Thanks! I will check out Doccano now m3at ↗ Thanks for building this, it's useful and well presented! tracyhenry ↗ Ah right now ranking is really rudimentary: I just use TextBlob's SA library to get a sentiment score for the comment, and combine it with the comment length. In theory we can also use BERT to get a sentiment score, which should be more accurate I guess? Love to hear your thoughts. m3at ↗ Simple is good! Especially for ranking as the objective is hard to define. Having looked at some samples on your site, imo it's good enough :) alanbernstein ↗ This is great. I just read Permutation City, which I coincidentally see recommended on HN all the time, so I was surprised not to see it in the search results or the top of the fiction or scifi lists. Any idea why that is? tracyhenry ↗ That might be that the book database I used is quite limited, sadly. Tycho ↗ Sounds good. Blocked by my work firewall though. ZeroGravitas ↗ If it was Oliver Sacks' Awakenings then it is non-fiction, though it did get turned into a movie. Tycho ↗ Different book - Kate Chopin Rd6n6 ↗ I don’t understand how software engineers get away with browsing the internet for fun at work when nobody else can themodelplumber ↗ Sounds more like personal development than fun? sillysaurusx ↗ That's a bit like saying watching porn is more for personal development than fun. Perhaps you'll learn something, but it's incidental. themodelplumber ↗ Is all that hyperbole really necessary? Each new sentence seems primed to leak edge and corner cases. Without giving more attention to such a rhetorical blind spot, I wonder how one could imagine they know the crux from the passenger side door. sillysaurusx ↗ Which sentence is mistaken? themodelplumber ↗ The one with all the generalizations sillysaurusx ↗ If it’s mistaken, it should be easy to explain why. Otherwise I’m inclined to believe it’s merely an uncomfortable truth. kritiko ↗ There’s often good intel on here that I have not been exposed to on other sources. Obviously spending 40 hours a week reading HN and getting into politics arguments on here is a waste, but there’s plenty of relevant news for most folks in tech if you stick to those topics. _y5hn ↗ Do you collect stats, how assertions line up with facts? Otherwise, what may seem likely and catchy, might just as well be opinionated, unsubstantiated and patchy. chadcmulligan ↗ https://xkcd.com/303/ [deleted] ↗ (comment deleted) personjerry ↗ The problem with reading book lists like this is that nobody has time to read all the books. That's a ton of crap out there and I want HN to help me filter through them. adewinter ↗ > With a deep learning model reducing all the reviews to "book names" you've successfully removed the value of the book discussions themselves. personjerry ↗ Oh, I was on mobile and could not see the comments section. It's interesting for sure. But what I want in particular is to learn why a book is ranked highly compared to other books. And I want to be sure that ranking is tailored to me, at a very, very high accuracy. lbriner ↗ They are ranked highly because of the number of times they are recomended in a comment. FredPret ↗ There's no way around the black box element of a book review, but Nassim Taleb suggests waiting a few decades and, if the book is still well known, then reading it. bachmeier ↗ Wow. What a helpful piece of advice (I guess he's smarter than the rest of us so it's hard to understand the genius of his strategy). Any mention of the cost of missing out on the content in the book for a few decades? dwighttk ↗ Just read older books until there aren’t any and then move onto newer ones. There are too many that are a couple decades old for you to ever run out. lifekaizen ↗ Love this question. I could imagine him suggesting reading academic papers for cutting edge things; like his 'barbell' excercise strategy of mostly walking with occasional HIITs. phgn ↗ The idea behind reading older books is that they're already proven to be useful - it's a filter for you to spend less time on useless information. It's generally called the "Lindy effect" dhosek ↗ I guess it's ok to read The C Programming Language, then. FredPret ↗ Technical manuals are more like a journal in the sense that you have to read all the new ones if you want to keep up. blondin ↗ the c programming language stood the test of time. tracyhenry ↗ > I want HN to help me filter through them. deaddabe ↗ Impressive work. tracyhenry ↗ I crawled about 20k books from Amazon. Thanks for pointing me openlibrary! nautilius ↗ Amazing and super useful: If I start reading today, and I read a book a day, it'll only take 112 years to finish, assuming that no additional books will be recommended in the next century. wrycoder ↗ I noticed that as I got several pages deep, the recommendations started to repeat. nautilius ↗ Then maybe only 30 years ;-) sushisource ↗ Heh, for a minute there I thought you meant Warhammer 40k books specifically, and I thought that was a pretty funny thing to be scraping from HN :) bnbond ↗ Same. I'm a little disappointed it's not. russellbeattie ↗ I'm pretty sure there's more Warhammer 40k books than there are days in the year... It's like someone heard the term "space opera" and thought that meant "soap opera in space". LanceH ↗ While I don't consider the 40k books on par with the better science fiction out there, I do enjoy that they bring a bit of scale and what it means to space. It's a different take from the rosy, post-scarcity, future of space. Bad things are really bad. Unattended good things turn bad on their own just from drift. KiranRao0 ↗ I've always considered 40k a satire of the entire sci-fi genre (and in many ways a satire on modern politics). In that way, I find it quite refreshing. And your statement of "unashamed embrace of over-the-top" resonates quite well. figassis ↗ This is amazing, thank you. defect0 ↗ Noticed an issue. Some, but not all, comments referencing Strunk and White's Elements of Style are showing up instead as Erin Gates' Elements of Style: Designing a Home & a Life tracyhenry ↗ Good catch! This is the limitation mentioned in my root comment - the algorithm will fail when two books have similar names. The partial solution is to look at authors too when available. Something to be included in the future. soheil ↗ "You're delusional. Where did I ever recommend reading Atlas Shrugged? Ayn Rand is nuts." bachmeier ↗ Interesting idea, but this is mentions of books, not recommendations. It includes comments by someone that's reading the book, has it on their reading list, or read it and thought it was terrible. tracyhenry ↗ The intention was to only show recommendations. But because of limited training data (we hand labeled ~4000 comments), the model wasn't able to filter out bad ones effectively. More training data should be able to solve it. jeron ↗ 40k good books out there and I can only read like 24 a year if I really push myself ehutch79 ↗ It has atlas shrugged in the top 10? gjm11 ↗ "Atlas Shrugged" is a polarizing book: people tend to either love it or hate it. And the people who love it love to tell other people how great it is, whereas many of the people who hate it just don't talk about it (because there's generally little need to talk about the badness of bad books). Kiro ↗ What's so strange about that? Objectivism used to be huge among hackers. hermitcrab ↗ Ugh. I'm not sure which is more offensive, the ideas behind it or the prose. bonniejawker ↗ are you planning to open source the app? could you do one for lobste.rs too? jp42 ↗ Slightly off from post. The best book recommendation I got is from one of the following ways, dedicated recommendation service or app never worked for me: spookyuser ↗ > random people mentioning book on platform like HN on a topic/post of my interest. rahimnathwani ↗ The most life-changing book recommendation I got was from HN: 'Teach your child to read in 100 easy lessons'. jp42 ↗ Thanks for the comment! My son is 3y8m. He knows letter and many word. Looks like this book could be what he needs to get on next level. TakerofVita ↗ Yeah, in my experience a lot of 'general' book reviews are super critical and don't really try to hook you. Going through several reviews, you just come away with the collected gripes and nitpicks of what is otherwise a good book. FinanceAnon ↗ Awesome idea and nice looking UI! I will definitely visit when I am looking for new books to read. tracyhenry ↗ I know this but I wasn't able to fix it. Would love suggestions on how to keep the scroll position in one div (for the books) but not the other (for the comments) when doing client-side navigation using Next.js... RheingoldRiver ↗ I was going to make this suggestion too, but now that I saw this comment, instead I'll suggest asking on the next.js subreddit? It seems somewhat active, maybe someone there can help - https://old.reddit.com/r/nextjs/ tracyhenry ↗ Although viewable on mobile, this app is best viewed on larger screens! :) zsmi ↗ It's a really interesting project. And I am sure it's really hard. tracyhenry ↗ It could be that I don't have this book in my book database. samuel ↗ This is amazing. Thank you! tracyhenry ↗ When I labeled the comments, I didn't label books that were criticized. So in theory the model should filter out negative reviews. But currently the training dataset is pretty limited in size so you still can see some negative ones. I suspect that with more training data this problem will go away. cyberge99 ↗ I’m Curious as to why you didn’t choose to monetize with affiliate links.
Regardless, fantastic work already!
A couple of thoughts:
* It would be great if each book were to have its own URL (for sharing).
* Consider allowing the search to allow author input, e.g. if I want to find the book 'Who' by Geoff Smart, the single-word title isn't specific enough to show that book at the top of the search results.
In the case of the book I searched 'Who', showing it in 4th position seemed about right.
And the problem with Stephenson is that's rarely succint so a bad book from him turns into a huge loss of time.
Most comments I’ve read here indicate the last part of the book is the most unpopular, so perhaps I should go back someday and read the middle.
Of course some might argue those are the best things about his books, but while Chryptonomicon is perhaps my favorite sci-fi novel I've read, I think it takes a certaim type of person to enjoy a book where the plot gets interupted for 5 page descriptions on how to eat Captain Crunch or a who fucked who of the Greek pantheon.
You mentioned transformers and BERT for large NLP models. I've been playing around with this too and it's a really powerful approach. Have you used spacy-transformers? [0]
The approach is pretty cool and can be used with BERT, GPT-2/Hugging Face etc.
I'm just starting to experiment with GPT-J and thinking of trying this approach also [1].
Anyway, totally awesome project and the results are really good. This stuff really is almost unreasonably effective!
[0] https://explosion.ai/blog/spacy-transformers
[1] https://6b.eleuther.ai/
Replies to your pinned top comment seems disabled, so allow me to ask here (sorry for the hijack):
> used a custom NLP-based ranking function to sort the comments
Can you expand on this function? (I'm familiar with NLP and most SOTA models)
Though if you wanted to try other things just for fun, maybe:
- Count matching NER, comments with a lot of book recommendations tend not to detail why they like the specific one currently filtered
- maybe down weight comments that are too short (after the matched NER is subtracted) as they seem to just have a title+link?
Not ML but while I'm here, on Android the comment section appear fine but has an horizontal scroll that seems spurious, with lots of blank space to the right (Galaxy S10, chrome & firefox)
A few years ago I found an article that was something like '100 short books everyone should read before they're 40'. It was a mix of fiction and non-fiction. I've never been able to find it again! But I really liked the list because these are books you can consume in a few hours and may be life changing.
I remember a few of the titles: Games People Play, Meditations, The Prince, The Art of War. (I suppose it may have been non-fiction only, although I think The Awakening may have been on there.)
Wish I could find the link again.
I've learned a lot from HN. But it wouldn't be good to fool myself into thinking that an employer wants to fund my personal development in this regard. Otherwise, they'd pay me to HN all day.
The crux of the issue is that it's impossible to work 8 hours every day. We all invent lies to fill the downtime.
Would your employer pay you to HN all day? If not, precisely how much of your day are they comfortable with you HN’ing? Are you sure it’s officially approved?
Waiting for Compiles is the usual, there's a lot of waiting in software - waiting for compiles, scripts to run, someone else to do something.
Thus the problem with existing solutions is NOT "limited recall" or "insufficient rules" or "no Amazon link".
And the problem with this "solution" is that there is no justification for why a book is great and applicable to my circumstances, and people have to trust your black box. Otherwise I'm likely to waste my time, just like reading books from any other crappy recommendation engine.
With a deep learning model reducing all the reviews to "book names" you've successfully removed the value of the book discussions themselves. Therefore, for me this engine and all similar engines are strictly worse than simply going through the actual big threads themselves, i.e. https://news.ycombinator.com/item?id=21900498
Edit: I've just seen the embedded comments by switching to a desktop browser. It's a nice addition. However, for me to make sure I'm not wasting my time going through arbitrary books and comments, I would need to know why a book is ranked highly compared to other books. And I want to be sure that ranking is tailored to me, at a very, very high accuracy.
It literally shows each comment in full that it extracted the book name from. It also includes a link to the comment in the original thread. What more could you possibly want?
Also occasionally break the rule for a book you want to read. It isn’t like that would kill you.
Novels, philosophies, and histories are works that can stand the test of time if they’re good enough
The comments panel show the actual recommendations. And the books are ranked by number of recommendations. Is this not enough?
What data source are you using for the books, authors and covers? I looked at OpenLibrary [1] but the covers are not the same, so I suppose it is something else? Maybe Amazon directly somehow?
[1] https://openlibrary.org/search?q=zero+to+one&mode=everything
Recommendations would include comments like, "This novel is really the one that ties the previous 37 books together." or "You might want to skip the next dozen books if you're squeamish about things that ooze."
Then there is there is the unashamed embrace of over-the-top in so many different ways.
Interesting that that's one of the recommendations.
I think a book list is more useful if it has some books in it that some love and some hate, rather than only books that no one minds very much. Maybe some of them will turn out to be ones I love.
(I happen not to be a Rand fan myself.)
- told by friend
- someone I admire read book and commented on it
- I'm working on some problem, during that exploration i came across books.
- random people mentioning book on platform like HN on a topic/post of my interest.
Same! Some of my favorite book recommendations have especially come from this one, I don't know why but a one line comment on a HN thread of "what book changed your life" has become my favorite way for discovering books.
I find that I get sold on a lot more when it is just a random single comment on some thread somewhere that focuses on a single aspect of a book.
If you can find a hyper specific subreddit/forum/etc. for a sub-genre you like, then you will spend more time reading books than reviews...
One thing that I've noticed is that when I select another book, the scrollbar in the comment section doesn't automatically scroll up.
I was curious how many times some common textbooks were mentioned but didn't find them via the search, which could be user error. But to give a specific example. None of the books in this comment thread were found:
https://news.ycombinator.com/item?id=19893447
Comment text like this: "CMOS VLSI Design: A Circuits and Systems Perspective (4th Edition)" by Weste and Harris
should've been caught, right?
Does it take into account negative reviews/comments? I have seen that Why we sleep is being recommended in the 6 months tab, but, while it was received with a lot of praise, it was soon critizised by others researchers in the field and I would expect that the HN crowd would have followed that trend.
Is seems simple and easily justifiable reward. I didn’t click the links, but hopefully you used smile.amazon for charity.
This is novel and useful. Thank you.