Reported a flaw to OpenAI that lets users peek at others' chat responses. Got an auto-reply on May 29th, radio silence since. Issue remains unpatched :(
Avoided their bug bounty due to permanent NDAs preventing disclosure even after fixes. Following standard 45-day disclosure window—users should avoid sharing sensitive data until this is resolved.
I believe it is extremely important to disclose that the ‘responses leaks’ you obtained did not originate from LLM models themselves, but rather through other insecure systems / in a more conventional manner.
Just to avoid yet another case of hallucinations outputs getting misinterpreted.
> A single misconfiguration can leak thousands of sensitive conversations in seconds. Treating privacy as an afterthought is untenable when the blast radius is this large.
Massive security bug, well spotted. It's like Bank of America showing other people my transactions, or Meta leaking my WhatsApp messages.
This raises some serious questions about security.
> The leaked responses show clear signs of being real conversations: they start with contextually appropriate replies, sometimes reference the original user question, appear in various languages, and maintain coherent conversational flow. This pattern is inconsistent with random model hallucinations but matches exactly what you'd expect from misdirected user sessions.
A model like GPT-4o can hallucinated responses that are indistinguishable from real user interactions. This is easy to confirm for yourself: just ask it to make one up.
I’m certainly willing to believe OpenAI leaks real user messages, but this is not proof of that claim.
Reminds me of a time I found a serious issue with mailgun. Messaged them, no reply. Had to spam their twitter to get a response. Basically you could have stolen tons of API keys from users without their knowledge and mailgun never disclosed it.
I could have actually gone to their office in person if I wanted to be pedantic but it actually seemed like a pretty weird office space lol.
> I am issuing this limited, non‑technical disclosure:
> No exploit code, proof‑of‑concept, or reproduction steps are included here.
Then why bother? I feel a bit cynical here, but if the goal is to get this fixed, they're not going to care unless it becomes a zero day and is given to the masses, otherwise it's going to quietly be exploitable by the few unsavory groups who know of it and will never be patched. Isn't the whole point of responsible disclosures to give them a time clock to get this situated before actual publication? Forgive me if I'm wrong, I haven't been in that field in a long time.
Hi all, I work on security at OpenAI. We have looked into this report and the model response does not contain outputs from any other users nor does it reflect a security vulnerability, compromise, or exploit.
The original report was that submitting a message close to (but not quite) 1500 seconds to the audio transcription API would result in weird, unrelated, off-topic responses that look like they might be replies to someone else’s query. This is not what’s happening. Our API has a bug where if the tokenization of the audio (which is not strictly correlated with the audio length) exceeds a limit, the entire input is truncated, and the model effectively receives a blank query. We’re working with our API team to get this fixed and to produce more useful error messages.
When the model receives an empty query, it generates a response by selecting one random token, then another (which is influenced by the first token), and another, and so on until it has completed a reply. It might seem odd that the responses are coherent, but this is a feature of how all LLM's work - each token that comes before influences the probability for the next token, and so the model generates a response containing words, phrases, code, etc. in a way that appears humanlike but in fact is solely a creation of the model. It’s just that in this case, the output started in a random (but likely) place and the responses were generated without any input. Our text models display the same behavior if you send an empty query, or you can try it yourself by directly sampling an open source model without any inputs.
We took a while to respond to this. Our goal is to provide a reasonable response to reports. If you have found a security vulnerability, we encourage you to report it via our bug bounty program: https://bugcrowd.com/engagements/openai.
Thank you, I made an update to the original post with your explanation, and because you stated that the output was a pure hallucination, I also attached one of them.
16 comments
[ 2.5 ms ] story [ 45.5 ms ] threadJust to avoid yet another case of hallucinations outputs getting misinterpreted.
Massive security bug, well spotted. It's like Bank of America showing other people my transactions, or Meta leaking my WhatsApp messages.
This raises some serious questions about security.
I certainly wouldn't sign an indefinite NDA for a chance to win:
Average payout: $836.36
openai should be grateful, after all, they want all information to be free
A model like GPT-4o can hallucinated responses that are indistinguishable from real user interactions. This is easy to confirm for yourself: just ask it to make one up.
I’m certainly willing to believe OpenAI leaks real user messages, but this is not proof of that claim.
I could have actually gone to their office in person if I wanted to be pedantic but it actually seemed like a pretty weird office space lol.
For real? At least doesn't match the one on https://keybase.io/requilence
The original report was that submitting a message close to (but not quite) 1500 seconds to the audio transcription API would result in weird, unrelated, off-topic responses that look like they might be replies to someone else’s query. This is not what’s happening. Our API has a bug where if the tokenization of the audio (which is not strictly correlated with the audio length) exceeds a limit, the entire input is truncated, and the model effectively receives a blank query. We’re working with our API team to get this fixed and to produce more useful error messages.
When the model receives an empty query, it generates a response by selecting one random token, then another (which is influenced by the first token), and another, and so on until it has completed a reply. It might seem odd that the responses are coherent, but this is a feature of how all LLM's work - each token that comes before influences the probability for the next token, and so the model generates a response containing words, phrases, code, etc. in a way that appears humanlike but in fact is solely a creation of the model. It’s just that in this case, the output started in a random (but likely) place and the responses were generated without any input. Our text models display the same behavior if you send an empty query, or you can try it yourself by directly sampling an open source model without any inputs.
We took a while to respond to this. Our goal is to provide a reasonable response to reports. If you have found a security vulnerability, we encourage you to report it via our bug bounty program: https://bugcrowd.com/engagements/openai.