Show HN: I scraped 3B Goodreads reviews to train a better recommendation model (book.sv)
For the past couple months I've been working on a website with two main features:
- https://book.sv - put in a list of books and get recommendations on what to read next from a model trained on over a billion reviews
- https://book.sv/intersect - put in a list of books and find the users on Goodreads who have read them all (if you don't want to be included in these results, you can opt-out here: https://book.sv/remove-my-data)
Technical info available here: https://book.sv/how-it-works
Note 1: If you only provide one or two books, the model doesn't have a lot to work with and may include a handful of somewhat unrelated popular books in the results. If you want recommendations based on just one book, click the "Similar" button next to the book after adding it to the input book list on the recommendations page.
Note 2: This is uncommon, but if you get an unexpected non-English titled book in the results, it is probably not a mistake and it very likely has an English edition. The "canonical" edition of a book I use for display is whatever one is the most popular, which is usually the English version, but this is not the case for all books, especially those by famous French or Russian authors.
127 comments
[ 2.7 ms ] story [ 96.6 ms ] thread1.Sid Meier’s Memoir!: A Life in Computer Games — Sid Meier 2.Source Code: My Beginnings — Bill Gates 3.Build: An Unorthodox Guide to Making Things Worth Making — Tony Fadell 4.Prince of Persia: The Journals — Jordan Mechner 5.A Theory of Fun for Game Design — Raph Koster 6.Ask Iwata: Words of Wisdom from Satoru Iwata, Nintendo’s Legendary CEO — Viz Media (Editor) 7.Control Freak: My Epic Adventure Making Video Games — Cliff Bleszinski 8.Once Upon Atari: How I Made History by Killing an Industry — Howard Scott Warshaw 9.Press Reset: Ruin and Recovery in the Video Game Industry — Jason Schreier 10.Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture — David Kushner
Any chance we could get an API going at some point? Are you planning to open source the work?
I'm interested in the scrapping of Goodreads too. I'm building a book metadata aggregation API and plan on building a scrapper for Goodreads, but I imagine using a data center IP address will be a problem very fast. Were you scrapping from your home network?
* UI - once someone clicks "Add" you really should remove that item from the suggested list - it's very confusing to still see it.
* Beam search / diversification -- Your system threw like 100 books at me of which I'd read 95 and heard of 2 of the other 3, so it worked for me as a predictor of what I'd read, but not so well for discovery.
I'd be interested in recommendations that pushed me into a new area, or gave me a surprising read. This is easier to do if you have a fairly complete list of what someone's read, I know. But off the top of my head, I'm imagining finding my eigenfriends, then finding books that are either controversial (very wide rating differences amongst my fellow readers) or possibly ghettoized, that is, some portion of similar readers also read this X or Y subject, but not all.
Anyway, thanks, this is fun! Hook up a VLM and let people take pictures of their bookshelf next.
This is how filmaffinity works, which is the best recommendation system I've tried. They have a group of several dozen 'soulmates', which are users with the most similar set of films seen and ratings given; recommendations are other stuff they also liked, and you get direct access to their lists.
>then finding books that are either controversial or possibly ghettoized
Naively, I’d say the surprises are going to be better if you filter more different friends, rather than more controversial books among your friends. As in “find me a person that’s like me only in some ways, tell me what they love”. Long term this method is much better at exposing you to new ideas rather than just finding your cliques holy wars.
It did suggest Murderbot Diaries (not on the input but a series I have read and did like) and an Adrian Tchaikovsky I hadn't read :).
One suggestion would be to make the search less strict on diacritics. Searching for popular cook J. Kenji López Alt was only successful if I entered the correct O.
Another thing I noticed is that it tends to recommend 2nd and 3rd books in a series, which is a bit so-so. If I add the first book in a series, I probably already read the whole series...
p.s. one idea: when you click [Add] on the recommended books list, it should remove it from that list
p.p.s. if there is a way to filter out the spam "Summary of ____" books, that would be good too
I would also really like a possibility to add negative signal. It did also recommend books that seemed interesting to me but I ultimately didn’t like.
Overall quite impressive.
https://book.sv/#2300585,644416
SASRec was released in 2018 just after transformer paper, and uses the same attention mechanism but different losses than LLMs. Any plans to upgrade to other item/user prediction models?
Sadly my experience with the book recommender isn't too great because of the 64 book limit. If I import either the most recent or least recent 64 book, 95% of the books it recommends to me are books I've read. Though it was helpful for spotting a few books I've read that I didn't log on Goodreads. Guess I'm pretty consistent.
Tigana, Hyperion, A Fire Upon the Deep, Blindsight, Moby Dick
and I got a list. Sure, read all that or wasn't interested for reasons, I added (only Neuromancer on initial recommendations):
Neuromancer, VALIS, Quantum Thief, Towing Jehovah.
List did not get more interesting.
Book recommendations are still kind of difficult.
Honestly this would finally be the web2.0 we all wanted & hoped for. It's against majesty that it's all captured owned user content that is legally captured by essentially public message boards/sites.