Once upon a time, Hansel and Gretel were walking through the woods when they met a Sleeping Beauty called Snow White. As they tried to wake Beauty, a naked Emperor walked in screaming "Off with his head" before a Big Bad Wolf started huffing and puffing.
All the books that they are scanning exist in the national libraries where they were published and have been for the past century or so.
Lets find a random conference book on Amazon... Genetics, Radiobiology and Radiology Proceedings, Mid-Western Conference. It was published in 1959. It's a rare book in that I can only find one copy of it on Amazon.
There's one copy onsite at the Library of Congress in the General Collections and another copy offsite. You can get a reading card for the Science and Business Reading Room ( https://www.loc.gov/research-centers/science-and-business/ ) and request that book and read it.
Every book that has been published, there's a copy of it somewhere. If it was published in the UK, it is in the British Library ( https://youtu.be/ZNVuIU6UUiM ).
The books are there.
What's happening is that these books are getting bought by AI companies and scanned. If they weren't bought by the AI company then... the book in the... I'm not even sure. It's in Portland... if inventory doesn't sell, it may instead get thrown out. Maybe this year, maybe in ten years - it costs money to have inventory that doesn't sell.
Libraries will do book sales of books that aren't checked out frequently. https://www.booksalefinder.com . Sometimes they're donated, sometimes it's library discards.
What do libraries do as they cull their collections?
> Some libraries will donate weeded materials to community resale shops, Goodwill, or other resale shops. Some will take all of your unwanted books without question, but many resale shops don’t have the capacity to accept the large number of books that libraries discard. If you do find a place to take them, you’ll still need to use staff or volunteer time to get them there.
> You’ll often hear this idea from well-meaning community members, and there are some fun crafts that can be made from discarded books. A quick online search will turn up many ideas for ways to use your books in crafts for kids, teens, and adults, including folded book art, blackout poetry, collage or prints made on book pages, wreaths and garlands made from book pages, and more. But even if you do a LOT of craft activities at your library, it’s unlikely that you could use up enough of your discards to even notice the difference.
Good memories of visiting used bookshops near Kyoto University with stacks upon stacks of obscure literary works and research material. Lots of interesting books about the Japanese language that were never digitized and I always left with 2-3 new books. So I'm not super happy about AI companies hoovering this all up and not making the scans available.
Our family volunteers at a nonprofit that moves a huge number of books. We take donations and run massive charity sales, clearing tens of thousands of books a month. Pricing works like a ladder: you try to sell a book for a couple of bucks, then for a dollar, then by the $5 bag, then for free, and you still end up with thousands of books nobody wants even at no cost. These used to go straight to pulp. Now they go to AI labs for scanning. Would we rather they were read, or at least owned, by someone? Yes. Is scanning better than turning them into toilet paper? Yes, even if only marginally.
I'm sure the Anthropics of the world have shredded the last known copies of plenty of titles. But nobody talks about the last known copies that get pulped every day simply because there are so many books nobody wants and storing them is expensive.
My wife loves bulk book hauls. Those places that we frequent, work like a literal permanent discount warehouse - books in high shelves, on pallets, everywhere. Often hundreds of issues of the same one.
But there is so many books there that no one want's to read. Hundreds of the same book lying there for months or years.
Same for public book-sharing "libraries" (small shelves that look like bird house, usually in parks etc). People really like them and there are many in my city, but most books there are products of a gone era and a gone mindset. No one want's that even for free.
We were taught respect for books, but not everything is worth preserving.
I didn't go this year but my local town library has a book sale where books go for something like $10/bag. I donate some books to them throughout the year. There are still a lot of books available on the last or second to last day of the sale. I'm sure a huge number get pulped.
A few days back, I spent some time going through a World Book encyclopaedia set that my parents had bought for me when I was 4. It was basically an expensive investment back then into what would turn into a lifelong reading habit.
Most of the information in it is practically worthless today unfortunately - Saddam Hussein still alive, as were Katharine Hepburn and King Hussein of Jordan, no mention of Ceres and Pluto was still a planet, and the Internet and software had barely a mention. Most articles had less information than a standard Wikipedia article.
That being said, the way information was presented in them still outshines anything one may find on the internet today. Even the simple elements - neat diagrams and relevant images, proper sectioning and organization of the text, footnotes to other relevant articles...
Long gone are the days when I would simply take a bowl of ice cream and a volume and just read it end to end.
Well, they could turn the bad faith story into a good faith story by making them available for everyone to download perhaps. (AI companies "saving" old books!) But that would require giving a s*t which they don't and that is the real problem imho.
Both Anthropic and Internet Archive have gotten sued into oblivion by publishing companies once, there's no way they're going to do something even more flagrantly illegal
I have a similar thought too each time I walk past piles of discount books.
I'm in the camp that perhaps it's healthy to not grasp onto every bit of information. that some artifacts dying a natural death is maybe just the way things are
> Now they go to AI labs for scanning. Would we rather they were read, or at least owned, by someone? Yes. Is scanning better than turning them into toilet paper? Yes, even if only marginally.
Yeah, now instead old books being turned into toilet paper, we'll get turned into toilet paper.
And Sam Altman will become richer than God, and isn't that what really matters?
But don't worry! You'll still have access to ChatGPT until your savings run out.
A lot of these books are reference titles that are outdated to the point of uselessness and/or weren't that great/interesting when they were new. Not many people have interest in or use for textbooks from the 50s.
Ah yes, the litteral destruction of culture and physical media for a centralised subscription service. I love the liberal world of techno enclosures of our new overlords, viva el free market economy.
Can someone explain why the old books could really be relevant. I get the pre-nuclear steel analogy, but why is this relevant given how much more modern texts exist. A few years ago, millions of yahoo groups were erased but now a few thousand books are what is needed to run a successful AI company? I mean, it can barely be about the information in those books (that would be very often outdated), but just for a little more text (with ever less marginal gain), what is the benefit?
Cognition is encoded in language, they weren't brain-damaged yet. There's better (real, not token-exchange) thinking, which LLMs can copy and reproduce in novel arrangements.
The information density is a lot less for millions of yahoo groups. They are far more likely to cover the same topics and not have new information in them.
Books are more likely to be about a specific topic or story or time or setting and be more information dense
They are the memories of the productive part of society. You can leaf through them and get the feel of what it was like. You don’t need most of your memories, personality, or core ideals to be productive to the State.
It sounds like it is about the information in the books. The titles they're looking for are all nonfiction. Not everything is on the internet, and just because it's a few years old doesn't mean it's outdated.
The internet basically never delivered on the promise of replacing textbooks or even education as a whole. Wikipedia sucks on many topics, has insane internal politics, and is a tertiary source by design (redigesting blogs and books), whereas textbooks are generally secondary.
I see. So it is much less about 150 year old fiction books, but more about 1980s science literature that was only ever printed five times. That makes a lot more sense than what the public debate seems to be about.
> but now a few thousand books are what is needed to run a successful AI company?
They're scanning millions of books.
It's the diversity of text that helps. One of the lessons we've learned is that more training data leads to better models. Even old books have different mixes of word sequences that will improve the model. The returns are diminishing, but when you have the pipeline set up to ingest it you might as well keep adding to the dataset.
> why the old books could really be relevant
> it can barely be about the information in those books (that would be very often outdated),
LOL, not being rude: have you ever read a book outside of what they forced you to read in school? Most old books are not O'reilly's manuals for Visual Studio 2014, they don't go out of date.
They are interesting to human beings for the same reason they are interesting to the labs. If it was just about quantity of text then the labs could generate text with the prev. gen model and use that alone to scale to the next model, there is something of immeasurable value contained in books (hint: it starts with an i and rhymes with bin formation).
Pretty much as it happened with vinyl records twenty years ago. I remember seeing photos of this guy somewhere in Brazil standing atop heaps of vinyl records which he had amassed with HDLR intention. Now books. Some of us hold on forever.
You are a moron: vinyl records came and went in about 25 years, books have been the engine of human progress for at least the last 3 millenia, the books being burned by these misanthropic lunatics are not available in any other medium, this is not about fascination with some particular mediumn of transmission, it's about the contents
Before you buy the apologia that these books are not valuable or interesting: if they weren't valuable or interesting the AI labs would not be spending billions of dollars to purchase and scan them. Yes, everyone has a personal anecdote about pallets of garbage books but I have three insights for you that you may not have because you don't read or sift through pallets of garbage books:
1) The labs don't want garbage books, they want interesting books that are rare and unique. They want high quality training data, random permutations of language style are fine, but what you want is unseen information, unseen patterns of thinking, unseen ideas.
2) Most pallet of books contains lots of valuable and interesting works, maybe 1-3% but sorting through them takes time, money, and energy, that's why the labs are starting to purchase by the pallet, it's because they already have a fully automated process so they can always beat any bookseller small or large on cost to find the books of interest and value in a pile.
3) Many of these pallets may sit for years before being sorted, and many of the books may sit for years before being sold, but these things actually do eventually happen, valuable books are found, and they eventually make their way to interested readers, this is the business model of used bookstores. Most books of value don't get destroyed or thrown away.
Destroying human art, knowledge, and culture is an essential part of the business plan for frontier labs, it is not enough to steal and regurgitate all the art and information in the world, you also want to make it inaccessible through any other means than the regurgitation machine. Don't expect the book burning to be an isolated incident, they are coming for every other form of stored human knowledge or art, and yes, unfortunately while scanning it they will have to destroy the original copy. And attacking the past is only the beginning.
I have sifted through everything from piled up junky independent book shops, to cast offs from research libraries, to dumpsters of end of life books. Most books really are not worth the paper they are printed on.
> Most books of value don't get destroyed or thrown away.
Of value to who? Most used bookstores are boutiques that over-curate and will happily refuse or recycle books that they deem are inferior/irrelevant. This is a big reason why I prefer Half Price Books over most any other used bookstore, they sell most everything.
> Most books really are not worth the paper they are printed on.
Yeah, but like I said in the comment you are replying to, somewhere around 1-5% of books are worth the paper they are printed on and far more, and those are exactly the books the labs are trying to scan and destroy, they are not buying pallets of books and scanning them in order to scan the june 1995 tv guide for the 800th time, they want unique, interesting, and high quality text, because that's what feeds pretraining.
(3) is doing almost all of the moral lifting, in that it assumes that relatively few of the books get pulped in the first place:
> 3) Many of these pallets may sit for years before being sorted, and many of the books may sit for years before being sold, but these things actually do eventually happen, valuable books are found, and they eventually make their way to interested readers, this is the business model of used bookstores. Most books of value don't get destroyed or thrown away.
This contradicts base reality, as the vast majority of the books in such cases were already queued to be pulped.
> Destroying human art, knowledge, and culture is an essential part of the business plan for frontier labs, it is not enough to steal and regurgitate all the art and information in the world, you also want to make it inaccessible through any other means than the regurgitation machine.
Even if Anthropic wanted to make the books publicly available as a form of moralwashing, they legally can't do that. Their current process rides the line of fair use, in that it obeys the requirement of medium shifting to be considered legally defensible.
Don't like the outcome? Change the law to allow for digital archival without requiring destruction.
AI firms buying and destroying the sources of knowledge is a weird way to get us to Fahrenheit 451 but maybe the USA will get there before it switches to metric after all.
once these books have all been scanned and destroyed, doesn't it mean later competitors (and free-as-in-foss alternatives) wont ever be able to catch up?
it seems really sad that these books data isnt open in the first place...
You'd think they could give the scans to archive.org or similar. Even if the books are in copyright, they won't be forever. We can't let our culture be destroyed like this.
65 comments
[ 1.9 ms ] story [ 57.0 ms ] threadLets find a random conference book on Amazon... Genetics, Radiobiology and Radiology Proceedings, Mid-Western Conference. It was published in 1959. It's a rare book in that I can only find one copy of it on Amazon.
https://www.amazon.com/Radiobiology-Radiology-Proceedings-Mi...
Lets go find it...
https://search.catalog.loc.gov/instances/e3fb7a94-2b25-5388-...
There's one copy onsite at the Library of Congress in the General Collections and another copy offsite. You can get a reading card for the Science and Business Reading Room ( https://www.loc.gov/research-centers/science-and-business/ ) and request that book and read it.
It also happens that my alma mater has a copy of the book too. https://search.library.wisc.edu/catalog/999554514302121 - it's in the stacks in Ebling Library.
Every book that has been published, there's a copy of it somewhere. If it was published in the UK, it is in the British Library ( https://youtu.be/ZNVuIU6UUiM ).
The books are there.
What's happening is that these books are getting bought by AI companies and scanned. If they weren't bought by the AI company then... the book in the... I'm not even sure. It's in Portland... if inventory doesn't sell, it may instead get thrown out. Maybe this year, maybe in ten years - it costs money to have inventory that doesn't sell.
Libraries will do book sales of books that aren't checked out frequently. https://www.booksalefinder.com . Sometimes they're donated, sometimes it's library discards.
What do libraries do as they cull their collections?
https://nwls.wislib.org/what-to-do-with-discarded-books/
> Some libraries will donate weeded materials to community resale shops, Goodwill, or other resale shops. Some will take all of your unwanted books without question, but many resale shops don’t have the capacity to accept the large number of books that libraries discard. If you do find a place to take them, you’ll still need to use staff or volunteer time to get them there.
> You’ll often hear this idea from well-meaning community members, and there are some fun crafts that can be made from discarded books. A quick online search will turn up many ideas for ways to use your books in crafts for kids, teens, and adults, including folded book art, blackout poetry, collage or prints made on book pages, wreaths and garlands made from book pages, and more. But even if you do a LOT of craft activities at your library, it’s unlikely that you could use up enough of your discards to even notice the difference.
For example... https://reallifeartist.wordpress.com/tag/book-carving/ or https:&...
I'm sure the Anthropics of the world have shredded the last known copies of plenty of titles. But nobody talks about the last known copies that get pulped every day simply because there are so many books nobody wants and storing them is expensive.
But there is so many books there that no one want's to read. Hundreds of the same book lying there for months or years.
Same for public book-sharing "libraries" (small shelves that look like bird house, usually in parks etc). People really like them and there are many in my city, but most books there are products of a gone era and a gone mindset. No one want's that even for free.
We were taught respect for books, but not everything is worth preserving.
Most of the information in it is practically worthless today unfortunately - Saddam Hussein still alive, as were Katharine Hepburn and King Hussein of Jordan, no mention of Ceres and Pluto was still a planet, and the Internet and software had barely a mention. Most articles had less information than a standard Wikipedia article.
That being said, the way information was presented in them still outshines anything one may find on the internet today. Even the simple elements - neat diagrams and relevant images, proper sectioning and organization of the text, footnotes to other relevant articles...
Long gone are the days when I would simply take a bowl of ice cream and a volume and just read it end to end.
I'm in the camp that perhaps it's healthy to not grasp onto every bit of information. that some artifacts dying a natural death is maybe just the way things are
Yeah, now instead old books being turned into toilet paper, we'll get turned into toilet paper.
And Sam Altman will become richer than God, and isn't that what really matters?
But don't worry! You'll still have access to ChatGPT until your savings run out.
scanning+ocr: 0.05 per page * 300 pgs: 15/bk
acquisition+disposal cost: 2.00/bk
total acq+scan+dispose = 425M
settlement value / bk: $3,000 [bartz v anthropic]
settlement probability, let's say 1%: 0.01
expected settlement / bk : $30
expected settlement total: 750M [industry-wide this is certainly an underestimate]
legal fees: 25% of settlement total: 187M
total 1.36B
+ operations, engineering, storage
Books are more likely to be about a specific topic or story or time or setting and be more information dense
I'm not convinced. I think you are under-weighing the massive volumes of stuff like self-help books, romance novels, etc.
I also guess that they're targeting languages that aren't tier one for them yet. Like, Japanese is probably a relatively small corpus for them.
They're scanning millions of books.
It's the diversity of text that helps. One of the lessons we've learned is that more training data leads to better models. Even old books have different mixes of word sequences that will improve the model. The returns are diminishing, but when you have the pipeline set up to ingest it you might as well keep adding to the dataset.
LOL, not being rude: have you ever read a book outside of what they forced you to read in school? Most old books are not O'reilly's manuals for Visual Studio 2014, they don't go out of date.
They are interesting to human beings for the same reason they are interesting to the labs. If it was just about quantity of text then the labs could generate text with the prev. gen model and use that alone to scale to the next model, there is something of immeasurable value contained in books (hint: it starts with an i and rhymes with bin formation).
There's a number of places that destroyed vast amounts of data in the wind down of ZIRP that probably regret it now.
Lots and lots of information is not online. You'd be surprised.
1) The labs don't want garbage books, they want interesting books that are rare and unique. They want high quality training data, random permutations of language style are fine, but what you want is unseen information, unseen patterns of thinking, unseen ideas.
2) Most pallet of books contains lots of valuable and interesting works, maybe 1-3% but sorting through them takes time, money, and energy, that's why the labs are starting to purchase by the pallet, it's because they already have a fully automated process so they can always beat any bookseller small or large on cost to find the books of interest and value in a pile.
3) Many of these pallets may sit for years before being sorted, and many of the books may sit for years before being sold, but these things actually do eventually happen, valuable books are found, and they eventually make their way to interested readers, this is the business model of used bookstores. Most books of value don't get destroyed or thrown away.
Destroying human art, knowledge, and culture is an essential part of the business plan for frontier labs, it is not enough to steal and regurgitate all the art and information in the world, you also want to make it inaccessible through any other means than the regurgitation machine. Don't expect the book burning to be an isolated incident, they are coming for every other form of stored human knowledge or art, and yes, unfortunately while scanning it they will have to destroy the original copy. And attacking the past is only the beginning.
> Most books of value don't get destroyed or thrown away.
Of value to who? Most used bookstores are boutiques that over-curate and will happily refuse or recycle books that they deem are inferior/irrelevant. This is a big reason why I prefer Half Price Books over most any other used bookstore, they sell most everything.
Yeah, but like I said in the comment you are replying to, somewhere around 1-5% of books are worth the paper they are printed on and far more, and those are exactly the books the labs are trying to scan and destroy, they are not buying pallets of books and scanning them in order to scan the june 1995 tv guide for the 800th time, they want unique, interesting, and high quality text, because that's what feeds pretraining.
> 3) Many of these pallets may sit for years before being sorted, and many of the books may sit for years before being sold, but these things actually do eventually happen, valuable books are found, and they eventually make their way to interested readers, this is the business model of used bookstores. Most books of value don't get destroyed or thrown away.
This contradicts base reality, as the vast majority of the books in such cases were already queued to be pulped.
> Destroying human art, knowledge, and culture is an essential part of the business plan for frontier labs, it is not enough to steal and regurgitate all the art and information in the world, you also want to make it inaccessible through any other means than the regurgitation machine.
Even if Anthropic wanted to make the books publicly available as a form of moralwashing, they legally can't do that. Their current process rides the line of fair use, in that it obeys the requirement of medium shifting to be considered legally defensible.
Don't like the outcome? Change the law to allow for digital archival without requiring destruction.
it seems really sad that these books data isnt open in the first place...