Presumably the AIcorps avoid calling it "libgen" because that would be just a bit too obvious, and now the journalists aren't calling it that to avoid the Streisand Effect.
But frankly, I can't see how Books2 could be anything other than libgen.
I guess they hope if people have enough they won't sue for things like this but they fail to see that the people suing are rich to a degree many of us cannot comprehend. UBI won't change their behaviour one bit.
I’m very bullish on technology progress and the future of generative AI, but at the same time I cannot fathom deciding to consume such a vast amount of pirated content (or even any pirated content) and then releasing a paid service off the back of it, and not expecting trouble down the line.
For an open source, non-commercial project it might be controversial, but commercialising the aggregate pirated content of 180,000 stolen books is pretty bold.
There is a very clear line between content that’s freely available (eg. scraping the web) and this. And even the web scraping is obviously controversial for some…
I didn’t intend for it to be commercialized. It’s a research contribution, nothing more.
It was also freely available via the web when I did it. I got all the books from https://the-eye.eu/public/Books. So it was literally web scraping.
In fact, you can, too. https://the-eye.eu/public/Books/ThoseBooks/ seems to have a wide selection, though it’s an order of magnitude less than their Bibliotik scrape. It’s interesting that no one cared until recently.
Just because it was web scraping doesn't mean it was OK to take it. You knew the sources contained copyrighted material and you did it anyway. I'm not trying to attack you but I don't see how saying it was 'research' makes it OK.
> I didn’t intend for it to be commercialized. It’s a research contribution, nothing more.
FWIW my comment is aimed at the likes of OpenAI who are now profiting off this work. For example, every book that I own (as in paid money for) and searched for is in books3. I pay OpenAI currently for ChatGPT Pro. I feel like I should be getting a discount...
I don't really have a strong opinion on the collation of material for research purposes as you did. LLMs are going to have a truly outsized impact on the world in the long term. But we do still need to consider the authors (individual creators) who are getting caught up in the crossfire of this knowledge/technology revolution.
The real tragedy is that people had to work so hard to figure out exactly what was in the dataset. I just never got around to releasing the metadata, which is here: https://battle.shawwn.com/books3-metadata.jsonl
Dozens of people have told me I belong in prison for making books3. And as far as I can tell, it’s had zero economic impact on authors.
If I do end up in prison, I think I’ll join the ranks of "people we look back on throughout history with morbid curiosity". There are lots of examples of people being imprisoned for things we now consider harmless. The buggery act was used to purge the Knights Templar.
See also https://annas-archive.org/llm for acquiring ~20M books. But at this point you’d better be willing to risk being despised at best.
At this point I don’t know why I even did anything in AI. llama-dl was DMCA’ed, books3 is prison fodder, and the big players do as they please with very little risk. Even if the lawsuits are successful, they’ll just pay out damages from VC money.
I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters. Hopefully they’ll continue to be able to.
By the way, the reporter here, Alex Reisner, is a wonderful fellow, and a fellow hacker. He has a bunch of open source projects.
> “as far as I can tell, it’s had zero economic impact on authors”
Why are you the right person in the world to make these decisions on the behalf of thousands of living authors?
I can understand distributing old works and abandonware. Copyright protection is far too long. But I don’t understand taking the recent creative work of somebody who’s alive and trying to make a living, and packaging it so that trillion-dollar corporations have a plausible excuse to use the work without permission.
> Why are you the right person in the world to make these decisions on the behalf of thousands of living authors?
This is a good argument. Why would anyone focus on the fact that no harm has been caused when we could be talking about theoretical harms that we can’t describe or quantify??
Can I start writing books in your name and selling them on Kindle using your reputation with made-up quotes from your professional associates? Why not? How can you quantify the harm?
The inalienable right to protect us from a computer using statistical math to output the word “embiggens” when given a Simpsons prompt is as obvious as free speech and liberty.
Edit
>Can I start writing books in your name and selling them on Kindle using your reputation with made-up quotes from your professional associates? Why not? How can you quantify the harm?
You make a very compelling point here. The books3 torrent has made so much money by doing what you’ve described. Quantifying the harm is easy, simply take all of the money from books3 (the file archive)’s ill-gotten-gains checking account and give it to its victims.
Excellent comparison because the implementation details of free speech are far from obvious and have been constantly tuned and tweaked over the centuries. (“Liberty” isn’t any specific right.)
It’s easy to pick an example that you consider benevolent and harmless. But have you considered that authors simply may not want their work included in a Simpsons meme generator?
Samuel Beckett dictated the set design for his plays to every last detail. Because of long copyright, you still can’t perform your own version of “En attendant Godot” today. I don’t think that’s necessarily a good thing. But I have to respect Beckett’s obsessive creative vision and that the law doesn’t let me (or Disney) make our own versions of Godot yet.
Re: your edit —
You wouldn’t have written the books I’m selling on Kindle in your name. So by the same logic that there’s no harm to authors when Meta and Google and OpenAI sell products derived from pirated books, there’s no harm to you when I use your name for my own ventures.
Sillysaurusx created a database whose only appeal is that it contains works by famous authors. This is not fundamentally different from me writing a book in your name.
> But have you considered that authors simply may not want their work included in a Simpsons meme generator?
Ironically copyright law seems to explicitly allow this under fair use for parody...
I totally understand that authors need to put food on the table, and economic rights are important. What I don't get is why we should worry that some author might be unhappy that their works end up in a meme. It seems to be the last thing we should worry about in the grand scheme of things.
> But have you considered that authors simply may not want their work included in a Simpsons meme generator?
“Liberty” isn’t a right. What is the word that you would use to describe the specific right to have your possible wishes be enforced as a matter of law?
I’m fascinated to learn that rights (which are universal) include the right to stop anyone from doing anything I don’t like.
Edit:
Wait what
> Samuel Beckett dictated the set design for his plays to every last detail. Because of long copyright, you still can’t perform your own version of “En attendant Godot” today. I don’t think that’s necessarily a good thing. But I have to respect Beckett’s obsessive creative vision and that the law doesn’t let me (or Disney) make our own versions of Godot yet.
You believe that your inability to perform Godot extends from a moral right? I am assuming you wouldn’t bring up moral rights and then immediately point out that you’re actually talking about your untested interpretation of a law that you don’t actually find to be just.
It includes “the right to the integrity of the work”, which is how the Beckett estate prevents me from making my own set design for “Godot” even if I pay for the rights to perform the piece.
That's shifting the goalposts. They're totally different things.
I know the comments I post here are probably going to end up in more than a dozen web scrape databases, and I'm personally fine with that. I'm not fine with somebody impersonate me and steal my identity.
Are there any allegations at all that the books3 database misrepresents any works of any authors? Not even the generative models do that -- if it was obvious that the models had a tendency to misattribute authorship of works, they'd have been sued for that (among other things) already.
You don't get to respond to "why would anyone focus on the fact that no harm has been caused when we could be talking about theoretical harms that we can’t describe or quantify??" by making up more fictional harms that you have no evidence happened.
> “Are there any allegations at all that the books3 database misrepresents any works of any authors?”
It’s a misrepresentation to include these works in a database when the authors didn’t choose to do so.
The database itself is a creative work. It’s directly using the reputation of thousands of other authors. Having your work included in another creative work without your approval is textbook infringement.
> It’s a misrepresentation to include these works in a database when the authors didn’t choose to do so.
That's not how any of this works. You can't opt-out from my database of "50 Books that are Overrated", or "50 Books that are Great" no matter how much you claim copyright.
There's a difference between "Having your work included in another creative work without your approval", and "writing books in your name and selling them on Kindle using your reputation".
I'm surprised that you seem to fail to see the distinction.
Or do you really think what happened to Jane Friedman (the author in that Ars article) is right?
Note how it was Amazon resisting her takedown request. The giant tech corporations love copyright infringement because it gives more power to them, as with the books3 case.
Correct. My intent was to acknowledge it was possible to do so (& is being done) not to give permission for it. :)
> Or do you really think what happened to Jane Friedman [...] is right?
Not at all.
> The giant tech corporations love copyright infringement because it gives more power to them [...]
I agree. One of the problems I have with the AI training situation is that corporations appear to be able to avoid complying with the onerous copyright regime they themselves created/support but demand individuals still adhere to it.
If they're not prepared to limit themselves to training data currently licensable to them or in the public domain maybe they shouldn't have supported copyright term extension/enforcement and/or work to ensure more works enter the public domain earlier.
https://spawning.ai seems to have the right idea. They propose creating an ai.txt standard. I’m also working with them to make an opt out for books3, for whatever that’s worth. Which given the current climate of “sue or jail anyone who disagrees” might only stoke the flames.
That's how copyright works. To create a derivative work, you need a license.
It's no more derivative than the works the authors produce, publicize and sell on their open source software platforms. Are they also going to be charged for every book released that has in some way contributed to their cultural hinterland?
We're talking about building a cultural context en-masse - not taking individual books to enjoy for free and plagiarise.
That single vs mass use of the books makes a significant ethical difference.
Assuming you believe technological advancements are a good thing, there's a balance to be struck between depriving authors of revenue they wouldn't have had in the first place (when you write a book in 2010, do you actually do so with the intent that you'd get paid by companies developing an AI with that data?), and stifling technological progress because there's new money to be made and we haven't figured how to divide the pie. (Yes, I believe they should be paid, but I don't think authors should morally be allowed to hold their works for ransom in the 'a million dollars per book, take it or leave it' sense. There should be some more reasonable thing here.)
We're all members of society, it's not like you want to live in an authoritarian environment where only Big Brother has the rights to make that decision. Mind you, according to your logic, even the authors of the books themselves would not have the right to make a general decision, only that of their own works. But they wouldn't have any insight at all to how allowing the data be trained on models will advance our understanding of AI tech.
Copyright is a deal with society. One might just as well ask why an author has any say over transactions between two independent third parties. The answer is because it has been deemed useful for the encouragement of creative works to provide for that say. It's not immutable and the rules can easily be changed if society thinks they should be.
It must be stressful being told you belong in prison, I hope you're doing OK.
FWIW, I think you -- and anyone else who collects, organises and makes this sort of stuff available -- are doing God's work. Archivists are heroes. It's the people who try to ruin the lives of those providing public goods that belong in prison.
Thanks. It is really stressful. I can’t even mention our newborn at this point without being told I’m a horrible father and that I only talk about her to manipulate public opinion. Meanwhile we just pulled up to the ER at 1am to find out why she’s not eating enough, and we have an echocardiogram in 7 hours to try to diagnose why her heart isn’t pumping enough.
The reason I reacted so strongly was out of genuine surprise for what they want here. The books3 dataset was public from day one. I’ve done interviews being totally transparent where the data came from. I also made it in my wife’s parents’ basement, so this letter seems to imply that their basement is under legal hold.
In other words, there is no data. There’s no evidence of anything. All the evidence was released openly — it’s how science is replicated.
So it seems like it’s a precursor for more actions in the future. And whatever those are, they’ll likely not be friendly.
Nobody cares until there’s someone’s making a billion dollars out of it.
Truth is your only mistake was that you made yourself traceable; someone would’ve done it anyway. Should’ve put the dataset on a magnet link and host via vpn.
> I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters.
Sounds like someone so focused on "if they could" but not giving much thought about "if they should".
I don't want you or any responsible ML researcher to go to prison. Responsible researchers make mistakes, but responsible researchers also consider the consequences of their actions. How much consideration was given to those? And given everything you know now, where would you draw the line? At what point do the consequences of this sort of research become indistinguishable from theft? Suppose the research enabled the piracy of a high fidelity copy of a single author's work without their permission? Would that be wrong? What about ~2e5 low fidelity copies of who knows how many man years of labor?
I’m not sure. Someone pointed out that I should’ve put a non commercial license on books3. But books3 isn’t my data to license.
As for actual impact, books3 made its way into BloombergGPT and LLaMA. I’m just happy it made LLaMA a little better.
Would I do it again? Well, given that there’s little benefit other than warm feelings about LLaMA and pride that hackers can make an open source ChatGPT competitor with this, that’s a tough question.
As for the line, I invite you to download books3 and try reading one of the books. The experience is positively awful. Notepad isn’t a good book reader; it’s missing images, which renders lots of the technical books less useful; and the markdown format isn’t even modern markdown syntax. Code snippets are indented with 4 spaces instead of surrounded with triple backticks.
Meanwhile anyone seriously interested in reading a book will go straight to libgen and get the real book themselves. So I’m skeptical of claims of harm.
As for whether it’s wrong, let history be my judge.
> As for the line, I invite you to download books3 and try reading one of the books.
I'm not going to and I suspect anyone who values their time won't either, especially when my public library meets my needs. But the line I'm asking you to look for isn't the one around a single book - it's the aggregate value previously locked in hundreds of thousands of them now released by the power of research like yours.
Even if you don't agree, I hope you appreciate that authors whose works are protected by copyright will feel entitled to a share of that value. Most of them didn't write their works to further ML research.
> Would I do it again? ... that’s a tough question.
I am reminded of when I was young how accessing an unprotected computer wasn't in itself a crime. People like me hacked into systems mainly out of curiosity. Then the media started making movies about teenagers starting nuclear wars with dial-up modems. Then journalists jumped on the hype train to sell their books and convinced people hackers needed to be punished. Then some went to jail and we got things like the CFAA. In a few short years what might have been wrong but wasn't explicitly illegal became so.
You are clearly a smart and intellectually honest person but society often treats such people poorly - especially when confronted by scary new technology - and the tone of your answers are giving me serious Aaron Swartz vibes. There's a lot of money at stake here.
It's fairly clear to me that trained models are a transformative rather than derivative works. However, without distributing forms which are not transformed, open/distributed/permissionless work in this space is either severely hampered, or basically impossible. I think a lot of people instinctively feel that removing barriers to their work is particularly important, since if we don't it could remain impossible for anyone outside a few megacorps -- who have the resource to obtain whatever they need in multiple ways without the additional liability of distributing non-transformed copies externally -- to build highly scaled up machine learning models.
I largely agree. I would also like to see more research into how to give authors incentives to license their works for transformative purposes such as training and ways to credit them when the results of inference draw disproportionately on a particular author's work.
I mean, copyright shouldn't be the infinite years it is currently. This used to belong to everyone collectively, and then publishers (not writers) decided they owned it.
Another big interest coming along now to decide that now they own it?
I don't care. Give people back their culture. 5 years of copyright is sufficient.
true. when it comes to technology, enjoy your 5 years of premium production. after that, you better have a good product, or the competition will eat you alive. as for creative works like books or movies or music, i'd go with 10 because it is more subjective type of work, creative instead of inventive and not technical and usually produced by individuals and not companies.
Following the same logic, everything you have archived, your wealth, your gadgets, your house, your car should belong to everybody after 10 years. Or do you want that only to apply to artists who (99% of them) dont make a lot of money anyways?
I agree that copyright has gotten out of hand, but 5 years is way too short. I can't imagine that anyone is going to spend YEARS writing a book, only for it to be public domain within 5. There needs to be commercial incentive for people to put in the effort to create things.
> I don't care. Give people back their culture. 5 years of copyright is sufficient.
This kind of attitude makes me a bit angry. I assume most people don't really understand how little most authors earn from their books. Same for most musicians, most indie app developers... artists...
If we keep treating copyright with so much disdain, there won't be any "culture" being created any more. It's barely a sustainable career as it is.
I wonder if it's because most people on HN are salaried employees, creating copyrighted work for someone else...
If you've spent any time as an individual creator, trying to make a living from your own copyrighted creations, I think you'd have a different attitude.
> I assume most people don't really understand how little most authors earn from their books. Same for most musicians, most indie app developers... artists...
But they don’t make so little because of too little copyright. They do so for a variety of reasons, including:
- people simply don’t buy their things, whether or not there is copyright on something with zero sales will not improve matter.
- strong copyright does not help authors/artists when the publishers require them to sign it all over. It’s like giving your bullied kid more lunch money, no amount will be enough if the bully simply takes it all.
Record companies are the ones that make most of the money off copyright. Artists profit from everything else (e.g., performances, sponsorships, merch, etc) which would be mostly unaffected by a shortened (maybe 10-20, not 5 years) copyright duration.
While it may be less common at this point, most Major Label recording contracts (for unproven acts) had 100% merch rights written into them throughout 80's and 90's.
I'm curious to see how this plays out and find it unfortunate that it happened.
Unfortunate because, I foresee the places where this information is shared between humans for the gain of knowledge are easier targets and will more than likely be targeted. I would be curious how an author might feel about that...their information being freely shared between humans for knowledge?
Curious because in my life experience, corporations don't have to play by the same rules as human beings and actually get rewarded for actions that would be admonished if an individual were to do them. So, even though corporate owned models were trained on and leverage this information for monetary gain (and not for the general idea of sharing knowledge), they will be glorified (as often seen here) for their monetization of this 'theft' and individuals will have a harder time gaining access to information.
One thing is for certain about this AI revolution, it has broken the open internet.
73 comments
[ 2.7 ms ] story [ 188 ms ] threadhttps://full-stack-search-prod.vercel.app/
Presumably the AIcorps avoid calling it "libgen" because that would be just a bit too obvious, and now the journalists aren't calling it that to avoid the Streisand Effect.
But frankly, I can't see how Books2 could be anything other than libgen.
What does that have to do with the article?
I’m very bullish on technology progress and the future of generative AI, but at the same time I cannot fathom deciding to consume such a vast amount of pirated content (or even any pirated content) and then releasing a paid service off the back of it, and not expecting trouble down the line.
For an open source, non-commercial project it might be controversial, but commercialising the aggregate pirated content of 180,000 stolen books is pretty bold.
There is a very clear line between content that’s freely available (eg. scraping the web) and this. And even the web scraping is obviously controversial for some…
It was also freely available via the web when I did it. I got all the books from https://the-eye.eu/public/Books. So it was literally web scraping.
In fact, you can, too. https://the-eye.eu/public/Books/ThoseBooks/ seems to have a wide selection, though it’s an order of magnitude less than their Bibliotik scrape. It’s interesting that no one cared until recently.
FWIW my comment is aimed at the likes of OpenAI who are now profiting off this work. For example, every book that I own (as in paid money for) and searched for is in books3. I pay OpenAI currently for ChatGPT Pro. I feel like I should be getting a discount...
I don't really have a strong opinion on the collation of material for research purposes as you did. LLMs are going to have a truly outsized impact on the world in the long term. But we do still need to consider the authors (individual creators) who are getting caught up in the crossfire of this knowledge/technology revolution.
Dozens of people have told me I belong in prison for making books3. And as far as I can tell, it’s had zero economic impact on authors.
If I do end up in prison, I think I’ll join the ranks of "people we look back on throughout history with morbid curiosity". There are lots of examples of people being imprisoned for things we now consider harmless. The buggery act was used to purge the Knights Templar.
See also https://annas-archive.org/llm for acquiring ~20M books. But at this point you’d better be willing to risk being despised at best.
At this point I don’t know why I even did anything in AI. llama-dl was DMCA’ed, books3 is prison fodder, and the big players do as they please with very little risk. Even if the lawsuits are successful, they’ll just pay out damages from VC money.
I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters. Hopefully they’ll continue to be able to.
By the way, the reporter here, Alex Reisner, is a wonderful fellow, and a fellow hacker. He has a bunch of open source projects.
Why are you the right person in the world to make these decisions on the behalf of thousands of living authors?
I can understand distributing old works and abandonware. Copyright protection is far too long. But I don’t understand taking the recent creative work of somebody who’s alive and trying to make a living, and packaging it so that trillion-dollar corporations have a plausible excuse to use the work without permission.
There will be zero economic impact on authors because they're already being screwed by publishers.
This is a good argument. Why would anyone focus on the fact that no harm has been caused when we could be talking about theoretical harms that we can’t describe or quantify??
Can I start writing books in your name and selling them on Kindle using your reputation with made-up quotes from your professional associates? Why not? How can you quantify the harm?
The inalienable right to protect us from a computer using statistical math to output the word “embiggens” when given a Simpsons prompt is as obvious as free speech and liberty.
Edit
>Can I start writing books in your name and selling them on Kindle using your reputation with made-up quotes from your professional associates? Why not? How can you quantify the harm?
You make a very compelling point here. The books3 torrent has made so much money by doing what you’ve described. Quantifying the harm is easy, simply take all of the money from books3 (the file archive)’s ill-gotten-gains checking account and give it to its victims.
It’s easy to pick an example that you consider benevolent and harmless. But have you considered that authors simply may not want their work included in a Simpsons meme generator?
Samuel Beckett dictated the set design for his plays to every last detail. Because of long copyright, you still can’t perform your own version of “En attendant Godot” today. I don’t think that’s necessarily a good thing. But I have to respect Beckett’s obsessive creative vision and that the law doesn’t let me (or Disney) make our own versions of Godot yet.
Re: your edit —
You wouldn’t have written the books I’m selling on Kindle in your name. So by the same logic that there’s no harm to authors when Meta and Google and OpenAI sell products derived from pirated books, there’s no harm to you when I use your name for my own ventures.
Sillysaurusx created a database whose only appeal is that it contains works by famous authors. This is not fundamentally different from me writing a book in your name.
Ironically copyright law seems to explicitly allow this under fair use for parody...
I totally understand that authors need to put food on the table, and economic rights are important. What I don't get is why we should worry that some author might be unhappy that their works end up in a meme. It seems to be the last thing we should worry about in the grand scheme of things.
see: 2 Live Crew's cover of Pretty Woman ending up in the supreme court
https://www.law.cornell.edu/supct/html/92-1292.ZO.html
> But have you considered that authors simply may not want their work included in a Simpsons meme generator?
“Liberty” isn’t a right. What is the word that you would use to describe the specific right to have your possible wishes be enforced as a matter of law?
I’m fascinated to learn that rights (which are universal) include the right to stop anyone from doing anything I don’t like.
Edit:
Wait what
> Samuel Beckett dictated the set design for his plays to every last detail. Because of long copyright, you still can’t perform your own version of “En attendant Godot” today. I don’t think that’s necessarily a good thing. But I have to respect Beckett’s obsessive creative vision and that the law doesn’t let me (or Disney) make our own versions of Godot yet.
You believe that your inability to perform Godot extends from a moral right? I am assuming you wouldn’t bring up moral rights and then immediately point out that you’re actually talking about your untested interpretation of a law that you don’t actually find to be just.
Yes, absolutely.
“Moral rights” in this context is a specific term:
https://en.wikipedia.org/wiki/Moral_rights
It includes “the right to the integrity of the work”, which is how the Beckett estate prevents me from making my own set design for “Godot” even if I pay for the rights to perform the piece.
Moral rights are dealt with differently in different jurisdictions.
I know the comments I post here are probably going to end up in more than a dozen web scrape databases, and I'm personally fine with that. I'm not fine with somebody impersonate me and steal my identity.
Are there any allegations at all that the books3 database misrepresents any works of any authors? Not even the generative models do that -- if it was obvious that the models had a tendency to misattribute authorship of works, they'd have been sued for that (among other things) already.
You don't get to respond to "why would anyone focus on the fact that no harm has been caused when we could be talking about theoretical harms that we can’t describe or quantify??" by making up more fictional harms that you have no evidence happened.
It’s a misrepresentation to include these works in a database when the authors didn’t choose to do so.
The database itself is a creative work. It’s directly using the reputation of thousands of other authors. Having your work included in another creative work without your approval is textbook infringement.
That's not how any of this works. You can't opt-out from my database of "50 Books that are Overrated", or "50 Books that are Great" no matter how much you claim copyright.
I'm surprised that you seem to fail to see the distinction.
Yes: https://arstechnica.com/information-technology/2023/08/autho...
Or do you really think what happened to Jane Friedman (the author in that Ars article) is right?
Note how it was Amazon resisting her takedown request. The giant tech corporations love copyright infringement because it gives more power to them, as with the books3 case.
Correct. My intent was to acknowledge it was possible to do so (& is being done) not to give permission for it. :)
> Or do you really think what happened to Jane Friedman [...] is right?
Not at all.
> The giant tech corporations love copyright infringement because it gives more power to them [...]
I agree. One of the problems I have with the AI training situation is that corporations appear to be able to avoid complying with the onerous copyright regime they themselves created/support but demand individuals still adhere to it.
If they're not prepared to limit themselves to training data currently licensable to them or in the public domain maybe they shouldn't have supported copyright term extension/enforcement and/or work to ensure more works enter the public domain earlier.
Everyone does it. And until the laws change, people will continue to.
You could make the same argument about LAION (and people do). The difference is that people have to scrape LAION themselves.
Google itself would be copyright infringement by that standard. They spider massive amounts of copyrighted content, and serve ads while doing so.
My goal was to make it possible for people like you and me to train a ChatGPT competitor.
That's how copyright works. To create a derivative work, you need a license.
> They spider massive amounts of copyrighted content
...while obeying `robots.txt`, which is the machine-readable version of "all rights reserved". They also accept DMCA notices.
It's no more derivative than the works the authors produce, publicize and sell on their open source software platforms. Are they also going to be charged for every book released that has in some way contributed to their cultural hinterland?
We're talking about building a cultural context en-masse - not taking individual books to enjoy for free and plagiarise.
That single vs mass use of the books makes a significant ethical difference.
We're all members of society, it's not like you want to live in an authoritarian environment where only Big Brother has the rights to make that decision. Mind you, according to your logic, even the authors of the books themselves would not have the right to make a general decision, only that of their own works. But they wouldn't have any insight at all to how allowing the data be trained on models will advance our understanding of AI tech.
FWIW, I think you -- and anyone else who collects, organises and makes this sort of stuff available -- are doing God's work. Archivists are heroes. It's the people who try to ruin the lives of those providing public goods that belong in prison.
sillysaurusx, I salute you.
The Silverman folks served me papers a few days ago. https://x.com/theshawwn/status/1704559992135717238?s=61&t=jQ...
I’ve made negative dollars from books3. Thus far it’s been an impressive exercise in self harm.
At this point I’m just living each day to the fullest. Whatever happens will happen. I’ve been making a thread of firsts to capture the magic moments. https://x.com/theshawwn/status/1706375699756384318?s=61&t=jQ...
I take solace in that people like you can benefit from my work. It was always intended for hackers.
You can let them know everything was lost in a backup mishap last year and have nothing to hold for them and they can stick it where it doesn't shine.
The reason I reacted so strongly was out of genuine surprise for what they want here. The books3 dataset was public from day one. I’ve done interviews being totally transparent where the data came from. I also made it in my wife’s parents’ basement, so this letter seems to imply that their basement is under legal hold.
In other words, there is no data. There’s no evidence of anything. All the evidence was released openly — it’s how science is replicated.
So it seems like it’s a precursor for more actions in the future. And whatever those are, they’ll likely not be friendly.
Truth is your only mistake was that you made yourself traceable; someone would’ve done it anyway. Should’ve put the dataset on a magnet link and host via vpn.
Sounds like someone so focused on "if they could" but not giving much thought about "if they should".
I don't want you or any responsible ML researcher to go to prison. Responsible researchers make mistakes, but responsible researchers also consider the consequences of their actions. How much consideration was given to those? And given everything you know now, where would you draw the line? At what point do the consequences of this sort of research become indistinguishable from theft? Suppose the research enabled the piracy of a high fidelity copy of a single author's work without their permission? Would that be wrong? What about ~2e5 low fidelity copies of who knows how many man years of labor?
As for actual impact, books3 made its way into BloombergGPT and LLaMA. I’m just happy it made LLaMA a little better.
Would I do it again? Well, given that there’s little benefit other than warm feelings about LLaMA and pride that hackers can make an open source ChatGPT competitor with this, that’s a tough question.
As for the line, I invite you to download books3 and try reading one of the books. The experience is positively awful. Notepad isn’t a good book reader; it’s missing images, which renders lots of the technical books less useful; and the markdown format isn’t even modern markdown syntax. Code snippets are indented with 4 spaces instead of surrounded with triple backticks.
Meanwhile anyone seriously interested in reading a book will go straight to libgen and get the real book themselves. So I’m skeptical of claims of harm.
As for whether it’s wrong, let history be my judge.
I'm not going to and I suspect anyone who values their time won't either, especially when my public library meets my needs. But the line I'm asking you to look for isn't the one around a single book - it's the aggregate value previously locked in hundreds of thousands of them now released by the power of research like yours.
Even if you don't agree, I hope you appreciate that authors whose works are protected by copyright will feel entitled to a share of that value. Most of them didn't write their works to further ML research.
> Would I do it again? ... that’s a tough question.
I am reminded of when I was young how accessing an unprotected computer wasn't in itself a crime. People like me hacked into systems mainly out of curiosity. Then the media started making movies about teenagers starting nuclear wars with dial-up modems. Then journalists jumped on the hype train to sell their books and convinced people hackers needed to be punished. Then some went to jail and we got things like the CFAA. In a few short years what might have been wrong but wasn't explicitly illegal became so.
You are clearly a smart and intellectually honest person but society often treats such people poorly - especially when confronted by scary new technology - and the tone of your answers are giving me serious Aaron Swartz vibes. There's a lot of money at stake here.
That doesn't matter. That's not your call to make without the permission of the copyright holders.
Another big interest coming along now to decide that now they own it?
I don't care. Give people back their culture. 5 years of copyright is sufficient.
> I can't imagine that anyone is going to spend YEARS writing a book.
Then you can't imagine why people write.
> There needs to be commercial incentive for people to put in the effort to create things.
No wonder you can't imagine why people write. How on earth did humanity invent writing before capitalism?
No. Look at A Game of Thrones as an example:
https://en.wikipedia.org/wiki/A_Game_of_Thrones
It was published in 1996. It became a bestseller in 2011 after the worldwide promotion it got due to the TV series.
This kind of attitude makes me a bit angry. I assume most people don't really understand how little most authors earn from their books. Same for most musicians, most indie app developers... artists...
If we keep treating copyright with so much disdain, there won't be any "culture" being created any more. It's barely a sustainable career as it is.
I wonder if it's because most people on HN are salaried employees, creating copyrighted work for someone else...
If you've spent any time as an individual creator, trying to make a living from your own copyrighted creations, I think you'd have a different attitude.
But they don’t make so little because of too little copyright. They do so for a variety of reasons, including:
- people simply don’t buy their things, whether or not there is copyright on something with zero sales will not improve matter.
- strong copyright does not help authors/artists when the publishers require them to sign it all over. It’s like giving your bullied kid more lunch money, no amount will be enough if the bully simply takes it all.
100% -> the label (just to be clear)
Unfortunate because, I foresee the places where this information is shared between humans for the gain of knowledge are easier targets and will more than likely be targeted. I would be curious how an author might feel about that...their information being freely shared between humans for knowledge?
Curious because in my life experience, corporations don't have to play by the same rules as human beings and actually get rewarded for actions that would be admonished if an individual were to do them. So, even though corporate owned models were trained on and leverage this information for monetary gain (and not for the general idea of sharing knowledge), they will be glorified (as often seen here) for their monetization of this 'theft' and individuals will have a harder time gaining access to information.
One thing is for certain about this AI revolution, it has broken the open internet.