Proprietary reasoning can be recovered from its encrypted traces. Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks to clients that can be replayed across sessions, users, and models. We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext, without ever attacking the stronger model directly or triggering its anti-distillation safeguards.
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace:
How to Steal Reasoning Without Reasoning Traces
https://arxiv.org/pdf/2603.07267
Trace Inversion is fascinating, but it’s more of an independent reconstruction that will give you some coherent-looking generated CoT; but not necessarily anywhere close or related to the underlying model’s CoT.
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
> I wonder if open models are going to start doing that too
Yes, some of them do do that. For example Moonshot tried to reward shorter reasoning traces in between Kimi-K2.6 and Kimi-K2.7 Code, and the latter has a mild caveman accent in its reasoning traces that the former lacks.
Qwen3.8-Max also has terse reasoning, but I don't remember this being the case for Qwen3.6 models I ran locally.
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
>But the real question is: Is it okay to steal from a thief's hoard?
How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
The real issue is that API providers log everything. OpenAI/Anthropic already capture full CoT traces in their logs — they just don't expose them. Distillation via API is just making explicit what they already have.
I was going to comment on that. This is clearly a vibe-coded webpage. It sort of smells like GPT to me, or at least front-end design. But the author clearly went back and forth to make it beautiful. This is not the first output he got.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
It wouldn't matter if it was stored only on their servers. As long as they offer the feature to downgrade a chat to a dumber model that can be jailbroken (and the downgrade keeps the reasoning), this trick works.
If CoT wasn't stateless and you instead just got a reference which pointed to the CoT stored on the lab servers, the same vulnerability would still exist. Since you just need a weaker jailbroken model to read a smarter model's CoT. This being stateless or not doesn't really matter.
The stateless part is also important for enterprise customers that require zero data retention.
(they could scope CoT access per model, but then users couldn't switch models mid-session)
This is a neat attack against those encrypted reasoning blocks you get back from APIs like OpenAI and Gemini and Anthropic:
> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext.
Should be easy for them to fix though: switch up the encryption key so it only works with the API for each specific model, rather than being shared across all of their models.
And indeed, the paper says it's been fixed by all three providers (though no news on how they fixed it):
> All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
So ... just a thought but could this be somewhat solved by, say if you were an LLM benchmark creator, using clever trickery?
Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.
Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.
I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.
I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.
btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)
Encryption is irrelevant here. Even if it was kept fully server side, the actual issue is that they allow starting a conversation in a strong model and continuing it in a weaker one. Disallowing that entirely would be a huge hit to user experience.
85 comments
[ 0.25 ms ] story [ 15.2 ms ] threadHa! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-...
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267
Yes, some of them do do that. For example Moonshot tried to reward shorter reasoning traces in between Kimi-K2.6 and Kimi-K2.7 Code, and the latter has a mild caveman accent in its reasoning traces that the former lacks.
Qwen3.8-Max also has terse reasoning, but I don't remember this being the case for Qwen3.6 models I ran locally.
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
I can’t fault them too much, as logit based distillation is extremely effective.
Very useful for making smaller models out of bigger open weight models.
But very interesting result.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Also really good for identifying any contradictions in your system prompt and context.
This is the kind of stuff I point to when people talk about AI slop. AI is just a tool. You're still the person who has to deliver the output and have some taste.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
you have a secret to keep that is read by the llm, and untrusted input that wants to exfiltrate it.
by hell or high water, the agent is gonna output that text
The stateless part is also important for enterprise customers that require zero data retention.
(they could scope CoT access per model, but then users couldn't switch models mid-session)
> We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, and recover the stronger model’s hidden reasoning in plaintext.
Should be easy for them to fix though: switch up the encryption key so it only works with the API for each specific model, rather than being shared across all of their models.
And indeed, the paper says it's been fixed by all three providers (though no news on how they fixed it):
> All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.
Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.
I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.
I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.
btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)