GPT-4 turbo has a context window of 128k tokens, so you can divide every book into a few chunks for initial summarization, then merge the results for the final summary!
pdftotext is a Linux utility to extract text from PDF files. Then you can easily cut the text file into pieces. For best summarization, it's best that the pieces overlap a bit, so that important paragraphs that lie on the boundaries are not lost.
For any book popular enough to have be put into GPT training data, you can get a summary without uploading it. The idea that these big, corporate models (their "intellectual property" if you will) have been trained on data they downloaded from ebook pirating sites... it's a real head shaker.
If these were scrappy start-ups looking to survive one can sort-of understand why they might bend the rules about copyright to train their models.
But we're talking about behemoth companies here, one of which deigns to make a pitch worth 7 Trillion dollars (10% of global GDP).
So... they're trying to rake in investment at levels that are unheard of in human history. Surpassing Apollo, the Manhattan project, the great pyramids of Egypt, the Great Wall of China, and anything else one might think of.
I definitely thought you meant that you asked it to write 5 bullet points that covered/summarized 14,501 books haha. Would love to see the output from that prompt.
Me too! I was really looking forward to seeing a summary of so many books. Was expecting to see something like:
* life is hard
* people can be evil
* war is inevitable
* everyone is afraid of dying
* humans persevere even in the face of insurmountable odds
There's a quote from Naval Ravikant that I really like (but now can't find), "All self-help books essentially boil down to 'prioritize long-term payoffs over short-term ones'". Haven't seen a single instance of that being incorrect.
Yeah, but this pairs well with that joke about being lost in a hot air balloon. The one about the answer being detailed, technically correct, and utterly useless.
I mean, if homo sapiens could reliably prioritize long-term payoffs over short-term ones, the world would look very different.
Same here! Love the idea behind this, but of course the big question is was one able to legally provide the content of 14,501 books to ChatGPT? Or there some database that has the content of all such books, and one can license there use?
I see that the first book reviewed is "Rich dad poor dad" by Robert Kiyosaki. He's a known grifter that has been wrong on nearly everything over the last 15 years. See https://news.ycombinator.com/item?id=38877126
It looks like a lot of these are self-help or other fluffy books.
It's funny because whenever I read these books I feel like someone just took a bunch of bullet points and then added anecdotes and fluff to make it fill a book. Herd you're reversing the process.
This is my feeling two. Many times the author tries to say the same thing 20 different times to fill a book. Many ot those books could have been a great 30 minute presentation.
> Many times the author tries to say the same thing 20 different times to fill a book. Many of those books could have been a great 30 minute presentation.
This is indeed how many, many of these books started -- as a $x minute presentation or an article.
I always try to find the source presentation or article and consume that before reading a book. It has served me well. You get the core ideas, concisely, without the fluff.
From the other extreme, I had recently asked GPT to do something similar with the text of Botchan (a famous literary work in Japan).
The bullet points it gave were were somewhat accurate at points, although I would say that the timeline and relationships wrong at some points. His childhood caretaker became a "woman he had a relationship with" later on, which was a complete mischaracterization. I think the regular version, sans the book text, might be more accurate (as it is probably based on Wikipedia / etc..)
If you're into savage debunking of junk self-help books, check out Michael Hobbs and Peter Shamshiri's podcast _If Books Could Kill_. Their obliteration of _Rich Dad, Poor Dad_ (the first book highlighted on this site) is one of their best episodes.
It's a lesson in putting things into personal perspective. When you stop to look there are always someone worse off than you and likewise better off. Be happy with the little coincidences that makes up life.
Ok, I'm getting it now. AI spam is really becoming a problem. I think the page should be much more upfront about the content being AI generated summaries.
This is a great idea! I'd love to see 3 levels of summary: 1) 5-bullet summary, 2) 5-paragraph summary and 3) 5-page summary so I can keep digging deeper if I'm interested.
Not sure if any of this goes against copyright law, though...
EDIT: I wonder what would be the outcome of summarize all 1500 books together in 5 bullet points (like a single summary for all books)
Did you actually ask for bullet points and ChatGPT gave you numbered points instead? Or did you just ask for 5 point summaries and it chose to use numbered points?
For the books I checked where I felt I could reasonably guess what topics would be covered and in what order, the 5 points seemed to cover things in the same order I expect the book covered them.
If that is the case numbered points would be more appropriate than bullet points, so if it was ChatGPT that decided on numbers vs. bullets it looks like it made the right call.
>I took a course in speed reading, learning to read straight down the middle of the page, and I was able to go through War and Peace in 20 minutes. It's about Russia.
53 comments
[ 4.3 ms ] story [ 106 ms ] threadhttps://aicopyright.substack.com/p/has-your-book-been-used-t...
But we're talking about behemoth companies here, one of which deigns to make a pitch worth 7 Trillion dollars (10% of global GDP).
So... they're trying to rake in investment at levels that are unheard of in human history. Surpassing Apollo, the Manhattan project, the great pyramids of Egypt, the Great Wall of China, and anything else one might think of.
And they're not paying for the books ???
I mean, if homo sapiens could reliably prioritize long-term payoffs over short-term ones, the world would look very different.
This could be easily implemented with langchain for example.
It's funny because whenever I read these books I feel like someone just took a bunch of bullet points and then added anecdotes and fluff to make it fill a book. Herd you're reversing the process.
I always try to find the source presentation or article and consume that before reading a book. It has served me well. You get the core ideas, concisely, without the fluff.
The bullet points it gave were were somewhat accurate at points, although I would say that the timeline and relationships wrong at some points. His childhood caretaker became a "woman he had a relationship with" later on, which was a complete mischaracterization. I think the regular version, sans the book text, might be more accurate (as it is probably based on Wikipedia / etc..)
https://twitter.com/BookReviewAi
I find the results very interesting. Like this "The Hitchhiker's Guide to the Galaxy" one.
https://twitter.com/BookReviewAi/status/1758162004529885573
The sentences could be longer if that helps.
Great little tool!
Not sure if any of this goes against copyright law, though...
EDIT: I wonder what would be the outcome of summarize all 1500 books together in 5 bullet points (like a single summary for all books)
Did you actually ask for bullet points and ChatGPT gave you numbered points instead? Or did you just ask for 5 point summaries and it chose to use numbered points?
For the books I checked where I felt I could reasonably guess what topics would be covered and in what order, the 5 points seemed to cover things in the same order I expect the book covered them.
If that is the case numbered points would be more appropriate than bullet points, so if it was ChatGPT that decided on numbers vs. bullets it looks like it made the right call.
—Woody Allen
And did you ask it first what subject would be interesting to HN readers?