So this won't be happening in the US, but in the EU:
"When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing. Watermarking will be applied at the model level, which means it will be present no matter which Claude product or surface the text comes from."
> Generated text will carry embedded watermarks, and generated files will include digitally signed provenance metadata where supported.
This should make it easier to catch cheaters who use Claude, right? Unless everyone runs their artifacts through some watermark and metadata sanitizer?
Those invisible spaces get wiped by the first sanitizer in any normal ide. Worse it'll instantly break parsing for configs like yaml where spaces are critical for structure
I have had a hunch for a while now that (in addition to these tools), Anthropic has actually leaned in to Claude's distinctive manner of writing since it makes the text more obviously AI generated and thus less susceptible to misuse.
That's not necessarily the same thing as a markov-style fingerprint but it could be a correlated factor.
It's pretty trivial to command it to not speak that way. That's one of the first things you should write into the prompt. What style you want it to write in. Make it use a very concise and dry academic style with no overt LLMisms, melodramatic or flowery language, or metacommentary.
I don’t like the idea of hacking a response to contain a watermark. I also don’t like the idea of false positives detections coming directly from Anthropic. If people read more AI generated content, people will probably start writing more in that style
The flip side of this is that if AI-generated content becomes reliably identifiable and carries a stigma, then people might deliberately change their styles to be more diverse and human.
One example I've seen are junior employees at my company deliberately adopting a lowercase/less punctuation writing style so as to stand apart from AI.
> When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
I'd like to know a lot more about how that works.
A lot of my interactions with Claude return pretty precise text. If I ask it to edit a project and refactor a specific function in several places I know exactly what I want to happen, it will NOT be OK if those refactors have some kind of weird pattern baked into their text to act as a watermark.
I guess this may be covered by this:
> Content generated by Claude may not carry a detectable mark if, for example: [...] The passage is very short, leaving too little text for a reliable signal;
I can tell you how: Claude produces a huge wall of text with jargon ridden bullshit and invented terms no human subject matter expert would seriously use and overuse.
>I'd like to know a lot more about how that works.
My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.
Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.
So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).
Off the top of my head I would have thought zero width characters (eg: U+200B, U+200C) making some unique identifier sprinkled in amongst the output. But obviously far from foolproof since they could simply be removed.
Scott Aaronson talks about his project at OpenAI to do this^
You can carefully select which pseudorandom number generator (prng) you use to be able to id text of a certain length. I expect there is some performance characteristic you have to manage since you're doing this on every inference, but once you do that it doesn't change the output in any meaningful way (the prng is still a statistically valid prng, it just happens to let you check if the output used that prng)
The point is that you can do this simply by swapping to a different RNG, which isn't noticeable to the end user, and while it changes the output, it's not any different from how using a different seed or being lumped in a different batch will change the output.
^ excerpt:
> So then to watermark, instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI. That won’t make any detectable difference to the end user, assuming the end user can’t distinguish the pseudorandom numbers from truly random ones. But now you can choose a pseudorandom function that secretly biases a certain score—a sum over a certain function g evaluated at each n-gram (sequence of n consecutive tokens), for some small n—which score you can also compute if you know the key for this pseudorandom function.
there are many simpler methods, for example you can have a tiny windowed transformer operating on the output text and all you do is alter certain words (that don't change meanings) to maximize its surprise. the tiny language model will have a special training regime to build up a somewhat unique view of the language.
we are talking about a 0.5 bit watermark here (existence). I would have zero confidence in being able to reliably remove such a watermark from pretty much any medium.
I think a lot of the examples below are projecting more complicated options, but it could also be something just as simple as using a word with a hyphen in it every prime-numbered sentence. Or any other "puzzle-y" pattern.
Short version: it's likely a SynthID style statistical watermarking scheme that Anthropic is using. There's likely no hidden unicode and no large stable green/red scheme.
Longer version: I found some really statistically significant outcomes when using Gloaguen et al.’s (https://www.sri.inf.ethz.ch/blog/probingsynthid) detection algorithm for SynthID, though unfortunately it seems like the models might be pre-biased to pass that test already. The difficult thing is that it's hard to conclusively know which models definitely already have watermarking enabled and if there are any models that don't have watermarking, which means that being able to get a highly positive statistical result is much harder.
You can take a look at the full writeup, but many of the clues from Anthropic's blog post, plus negative results on many of the simpler tests, point to them using such a scheme. Of course, it's possible that they came up with a novel new scheme/algorithm, but amongst the available literature, the SynthID family of models is probably the best bet.
> We’re also working to enable users and other third parties to detect Claude’s embedded watermarks and provenance metadata.
This seems to be similar in execution to Google's SynthID. I hope they release actual code the technically proficient can use, unlike SynthID which can only (afaik) be queried with Gemini's UI.
If Claude was the only model family they could ship a change like this and users who want to cheat (or don't like watermarks for other reasons) would just have to put up with it.
In a world with many different competing models, the risk of losing customers to other providers over this is much more real.
Maybe they've looked at the numbers and the portion of people who clearly use Claude to cheat on examples etc is so tiny that losing them to other providers isn't a problem?
I’m more worried that this will degrade performance. I want the best results from a model, not the results that fit a constraint that’s not defined by me. Any increased cost or latency is also unacceptable.
Scott Aaronson spoke about this in a colloquium where he said that this was mooted at OpenAI before the decision was made by Altman to not implement it for the reasons you describe.
I am cancelling my Claude max 5x subscription and moving to ChatGPT pro. I have difficulty enough trying to ensure my meaning comes through correctly, along with everything else; to now have to look out for/analyse watermarks too?
I feel shamed enough by society, thanks Anthropic.
How exactly does this impact proofreading? You can manually apply the suggestions (typo here, unnatural sounding sentence there, etc.) the LLM gives you to your own content, and it would stay watermark-free.
Unless with "proofreading" you actually mean having the LLM write your content for you.
I've heard that this kind of watermarking process works by biassing the statistical sampling towards a partition of the set of possible next tokens (red set and green set), at each position. It might only be a slight nudge each time, but over a sequence of tokens, the likelihood of repeating the bias by chance is increasingly improbable.
The bias is different for each position and follows a defined RNG, seeded somehow predictably.
Can be either an open algorithm, or not. If not open, then an API could be provided to determine if text is watermarked or not.
How it applies to code - maybe it could be a subtle nudge to symbol names, etc, I'm just speculating (I only read about this in passing very recently).
Can someone help me understand how exactly this watermarking of text works?
Given that text is, well, text, and not some kind of binary format, I don't see how any watermarking can work unless you insert characters which are invalid under Unicode. I further don't really understand how this won't be perceivable by assistive technology (the "watermark" will just appear as either unreadable characters, or if the watermark is mixed thoroughly enough into the text, it will scramble the text to any speech synthesizer and will make it really really obvious). Thus, I don't see how this wouldn't be insanely trivial to remove. And this is before we get into things being put on the clipboard. Sure, I can press the "Copy" button at the end of each response, but what I can also do is manually select the response and copy it, or only copy partial selections, or any number of other things. How does this "watermark" (or any "watermark" technology) take into account this?
So, really, to summarize this: I see no way of this actually being technologically achievable unless we revise the very core of how computers work and encodings for textual information. So I'm very curious as to how this is actually supposed to work.
It's important to distinguish "watermark" from "signature" (and the text/binary distinction is not relevant here):
Signatures are extra data, added out-of-band to the existing data. Out of band data, by definition, is easily detected and stripped, so the utility of a signature is that authoring one requires secret knowledge. Philosophically, the presence of a signature is a kind of authentication, a desirable thing that is hard to grant and easy to revoke (the smallest change to the data renders it invalid).
Now, watermarks: if you flip it round and say you want to glue on a piece of undesirable data - something that represents disauthentication, like a cursed black spot of written-by-LLM - then you want it to resist removal efforts. And now right away you have a hard problem because your sticky data must be in band, or else it is trivially stripped. Not only that, in fact, it has to look enough like real signal that it isn't easily filtered. And on top of that, you can't distort the real signal too much, or people will complain. So you're cornered into doing a kind of steganography - hiding small amounts of information in the entropy, biasing the signal in perceptually plausible ways that are detectable to those in the know. Cartographers add fake streets ("trap streets") to catch plaigiarists - for LLMs, watermarking might take the form of subtly odd word choices.
So my code that Claude makes, which previously was using the best (most probable) tokens for the job, will now be getting worse in random positions, to appease a voluntary EU suggestion. Love that.
If the western AI companies are forced to comply with this type of BS, and develop their models to do their job while balancing a book on their head and hopping on one foot, the Chinese models just got a free pass to completely dominate the frontier.
Does it mean their models will always write slop? Making the writing non-collapsed to specific patterns seems to break any injected/learned fingerprinting.
146 comments
[ 0.40 ms ] story [ 62.3 ms ] thread"When a supported Claude model generates text, it weaves an imperceptible watermark directly into the text itself. You won’t see it, and it doesn’t change the meaning, quality, or readability of Claude’s response.
Because the watermark is part of the text, it will travel with the text when it’s copied and pasted elsewhere, and may persist through some editing. Watermarking will be applied at the model level, which means it will be present no matter which Claude product or surface the text comes from."
I suspect they'll roll out the watermark everywhere.
This should make it easier to catch cheaters who use Claude, right? Unless everyone runs their artifacts through some watermark and metadata sanitizer?
Can it be circumvented? Of course. Will most people go through the trouble to circumvent it? No.
That's not necessarily the same thing as a markov-style fingerprint but it could be a correlated factor.
One example I've seen are junior employees at my company deliberately adopting a lowercase/less punctuation writing style so as to stand apart from AI.
I'd like to know a lot more about how that works.
A lot of my interactions with Claude return pretty precise text. If I ask it to edit a project and refactor a specific function in several places I know exactly what I want to happen, it will NOT be OK if those refactors have some kind of weird pattern baked into their text to act as a watermark.
I guess this may be covered by this:
> Content generated by Claude may not carry a detectable mark if, for example: [...] The passage is very short, leaving too little text for a reliable signal;
public abstract class BaseAnimalBeanFactoryGeneratedFromClaudeFactory
My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.
Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.
So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).
https://www.pcmag.com/news/genius-we-caught-google-red-hande...
You can carefully select which pseudorandom number generator (prng) you use to be able to id text of a certain length. I expect there is some performance characteristic you have to manage since you're doing this on every inference, but once you do that it doesn't change the output in any meaningful way (the prng is still a statistically valid prng, it just happens to let you check if the output used that prng)
The point is that you can do this simply by swapping to a different RNG, which isn't noticeable to the end user, and while it changes the output, it's not any different from how using a different seed or being lumped in a different batch will change the output.
^ excerpt:
> So then to watermark, instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI. That won’t make any detectable difference to the end user, assuming the end user can’t distinguish the pseudorandom numbers from truly random ones. But now you can choose a pseudorandom function that secretly biases a certain score—a sum over a certain function g evaluated at each n-gram (sequence of n consecutive tokens), for some small n—which score you can also compute if you know the key for this pseudorandom function.
there are many simpler methods, for example you can have a tiny windowed transformer operating on the output text and all you do is alter certain words (that don't change meanings) to maximize its surprise. the tiny language model will have a special training regime to build up a somewhat unique view of the language.
we are talking about a 0.5 bit watermark here (existence). I would have zero confidence in being able to reliably remove such a watermark from pretty much any medium.
Short version: it's likely a SynthID style statistical watermarking scheme that Anthropic is using. There's likely no hidden unicode and no large stable green/red scheme.
Longer version: I found some really statistically significant outcomes when using Gloaguen et al.’s (https://www.sri.inf.ethz.ch/blog/probingsynthid) detection algorithm for SynthID, though unfortunately it seems like the models might be pre-biased to pass that test already. The difficult thing is that it's hard to conclusively know which models definitely already have watermarking enabled and if there are any models that don't have watermarking, which means that being able to get a highly positive statistical result is much harder.
You can take a look at the full writeup, but many of the clues from Anthropic's blog post, plus negative results on many of the simpler tests, point to them using such a scheme. Of course, it's possible that they came up with a novel new scheme/algorithm, but amongst the available literature, the SynthID family of models is probably the best bet.
This seems to be similar in execution to Google's SynthID. I hope they release actual code the technically proficient can use, unlike SynthID which can only (afaik) be queried with Gemini's UI.
If Claude was the only model family they could ship a change like this and users who want to cheat (or don't like watermarks for other reasons) would just have to put up with it.
In a world with many different competing models, the risk of losing customers to other providers over this is much more real.
Maybe they've looked at the numbers and the portion of people who clearly use Claude to cheat on examples etc is so tiny that losing them to other providers isn't a problem?
https://youtu.be/9udWn1Hlj_s?si=VWOiK5-y4zcyDoHI
I am cancelling my Claude max 5x subscription and moving to ChatGPT pro. I have difficulty enough trying to ensure my meaning comes through correctly, along with everything else; to now have to look out for/analyse watermarks too?
I feel shamed enough by society, thanks Anthropic.
Unless with "proofreading" you actually mean having the LLM write your content for you.
The bias is different for each position and follows a defined RNG, seeded somehow predictably.
Can be either an open algorithm, or not. If not open, then an API could be provided to determine if text is watermarked or not.
How it applies to code - maybe it could be a subtle nudge to symbol names, etc, I'm just speculating (I only read about this in passing very recently).
Given that text is, well, text, and not some kind of binary format, I don't see how any watermarking can work unless you insert characters which are invalid under Unicode. I further don't really understand how this won't be perceivable by assistive technology (the "watermark" will just appear as either unreadable characters, or if the watermark is mixed thoroughly enough into the text, it will scramble the text to any speech synthesizer and will make it really really obvious). Thus, I don't see how this wouldn't be insanely trivial to remove. And this is before we get into things being put on the clipboard. Sure, I can press the "Copy" button at the end of each response, but what I can also do is manually select the response and copy it, or only copy partial selections, or any number of other things. How does this "watermark" (or any "watermark" technology) take into account this?
So, really, to summarize this: I see no way of this actually being technologically achievable unless we revise the very core of how computers work and encodings for textual information. So I'm very curious as to how this is actually supposed to work.
Signatures are extra data, added out-of-band to the existing data. Out of band data, by definition, is easily detected and stripped, so the utility of a signature is that authoring one requires secret knowledge. Philosophically, the presence of a signature is a kind of authentication, a desirable thing that is hard to grant and easy to revoke (the smallest change to the data renders it invalid).
Now, watermarks: if you flip it round and say you want to glue on a piece of undesirable data - something that represents disauthentication, like a cursed black spot of written-by-LLM - then you want it to resist removal efforts. And now right away you have a hard problem because your sticky data must be in band, or else it is trivially stripped. Not only that, in fact, it has to look enough like real signal that it isn't easily filtered. And on top of that, you can't distort the real signal too much, or people will complain. So you're cornered into doing a kind of steganography - hiding small amounts of information in the entropy, biasing the signal in perceptually plausible ways that are detectable to those in the know. Cartographers add fake streets ("trap streets") to catch plaigiarists - for LLMs, watermarking might take the form of subtly odd word choices.
They should make it easier, to detect slop so we can ignore it quickly.
I hope Pangram makes an API or an extension to analyze a page to detect slop on a page and then closes the tab immediately.
Nobody should be wasting time on garbage LLM output in code, text, image and videos.
EU regulation does it again!