How can I prevent my site from being a free dataset for LLMs?
Hi I am a blogger working on a small niche . I have written all these articles from the ground up with considerable effort. I don't want this to end up just as a free training data set for LLMs. Is there anything I can do to prevent that and still keep my site open free for visitors?
65 comments
[ 3.5 ms ] story [ 139 ms ] threadhttps://duckduckgo.com/?t=ffcm&q=cat+fud&iax=images&ia=image...
For others, take same measures as unwanted traffic or scrapping.
In any case, OpenAI should inspect every website's terms of use before ingesting it in their training data. They shouldn't be exempted from this work. We shouldn't have to conform to their methods, there are laws and systems in place for that. Expensive, yes.
If you want to use my work, credit me.
Try and tell Disney “if you don’t want your media pirated of copyright infringed, don’t publish anything”.
"your" AI ?
> If you don't want AI learning from your work, then don't publish it.
If you don't want me stealing and reusing your licensed open source code don't make it public
If you don't want me to steal your car don't park it on public roads
See how dumb that is ?
A practical matter, larger point completely aside: a nonzero number of individuals and corps will indeed use licensed code internally if they come across it and they feel it helps their goals.
For Common Crawl, the documentation says blocking it on robots.txt should work, as for Wikipedia, Reddit, and books, there's no option than to not participate AFAIK.
OpenWebText2 has no mention of robots.txt, so good luck with that.
Except if you are forcing people for giving a phone number for viewing a blog post , and want that china thing where if you not smile walking through a gate, your credits go down.
Edit: let be enough a third party service (namely a capcha, which uses my general "login") which assures a site that I am not a bot.
might as well poison the well
that will stop any kind of robot, AI powered or not, from scraping your site /s
Most bloggers with specialized knowledge do not write everything on the blog. The blog can be a summary or highlights of something bigger, like a research paper or a book.
Most "knowledge workers" are not making their income by writing online. They are parlaying that into consulting projects or speaking gigs, things an LLM can't replace.
You want visitors to be able to freely benefit from your work. What's wrong with AI also benefiting? Or more specifically the AI's eventual users?
If you read copywritten material and then pass it off as your own you are plagiarizing. Words in a dictionary don't come under that, but I'd bet that if you released a new dictionary that was mostly copied from the old one, most people would consider that plagiarism as well.
I haven't seen any service copy out large block of text enough to make me think it's reasonable to call their output plagiarized.
Meaning, if the LLM I use will only repeat an idea that many someone's have written about, such that it's seen the idea, or parts of that idea many times. Why is that still plagiarism? Or rather, worthy of direct attribution? Or why was I wrong to use the argument about citing a dictionary here?
(I'm aware that a number of people are working on giving memory so AI can quote from pages like wikipedia. But I don't think it's fair to call that "training data")
As you go narrower with a query only one source of truth is available at that point it does plagiarize.
Have you ever seen chatgpt cite or credit it's sources?