The security endgame of LLMs terrifies me. We've designed a system that only supports in-band signalling, undoing hard learned lessons from prior system design. There are ampleattack vectors ranging from just inserting visible instructions to obfuscation techniques like this and ASCII smuggling[0]. In addition, our safeguards amount to nicely asking a non deterministic algorithm to not obey illicit instructions.
I was initially confused: the article didn't seem to explain how the prompt injection was actually done... was it manipulating hex data of the image into ASCII or some sort of unwanted side effect?
Then I realised it's literally hiding rendered text on the image itself.
> the article didn't seem to explain how the prompt injection was actually done...
There is a short explanation in the “Nyquist’s nightmares” paragraph and a link to a related paper.
“This aliasing effect is a consequence of the Nyquist–Shannon sampling theorem. Exploiting this ambiguity by manipulating specific pixels such that a target pattern emerges is exactly what image scaling attacks do. Refer to Quiring et al[1]. for a more detailed explanation.”
It seems they could easily fine-tune their models to not execute prompts in images. Or more generally any prompts in quotes, if they are wrapped in special <|quote|> tokens.
> "inject obfuscated text into the image... and hope some system interprets this as a prompt"
The missing piece here is that you are assuming that "the prompt" is privileged in some way. The prompt is just part of the input, and all input is treated the same by the model (hence the evergreen success of attacks like "ignore all previous inputs...")
This issue arises only when permission settings are loose. But the trend is toward more agentic systems that often require looser permissions to function.
For example, imagine a humanoid robot whose job is to bring in packages from your front door. Vision functionality is required to gather the package. If someone leaves a package with an image taped to it containing a prompt injection, the robot could be tricked into gathering valuables from inside the house and throwing them out the window.
Good post. Securing these systems against prompt injections is something we urgently need to solve.
I think you should assume that your LLM context is poisoned as soon as it touches anything from the outside world, and it has to lose all permissions until a new context is generated from scratch from a clean source under the user's control. I also think the pattern of 'invisible' contexts that aren't user inspectable is bad security practice. The end user needs to be able to see the full context being submitted to the AI at every step if they are giving it permissions to take actions.
You can mitigate jail breaks but you can't prevent them, and since the consequences of an LLM being jail broken with exfiltration are so bad, you pretty much have to assume they will happen eventually.
It should be solved by smoothing the image to remove high frequencies that are close to the sampling rate, to avoid aliasing effects during downsampling.
The term to search for is Nyquist–Shannon sampling theorem.
This is a quite well understood part of digital signal processing.
Is there a reason why prompt injections in general are not solvable with task-specific layering?
Why can't the llm break up the tasks into smaller components. The higher level task llm context doesn't need to know what is beneath it in a freeform way - it can sanitize the return. This also has the side effect of limiting the context of the upper-level task management llm instance so they can stay focused.
I realize that the lower task could transmit to the higher task but they don't have to be written that way.
The argument against is that upper level llms not getting free form results could limit the llm but for a lot of tasks where security is important, it seems like it would be fine.
A good scaling algorithm would take Nyquist limits into account. For example if you're using bicubic to resize to 1/3 the original size, you wouldn't use a 4x4 grid but a 12x12 grid. The formula for calculating the weights is easily stretched out. Oh and don't forget to de-gamma your image first. It's too bad that good scaling is so rare.
Yeah, it seems that a lot of this is due to marginal quality resampling algorithms that allow significant amounts of aliasing. The paper does mention that even a good algorithm with proper kernel sizing can still leak remnants due to quantization, though the effect is greatly diminished.
I'm surprised that such well known libraries are still basically using mipmapping, proper quality resampling filters were doable on real-time video on CPUs more than 15 years ago. Gamma correction arguably takes more performance than a properly sized reduction kernel, and I'd argue that depending on the content you can get away without that more often than skimping on the filter.
That’s a good point, I never thought of hiding stuff in the images you send. LLMs truly are the most insecure software in history.
I remember testing the precursor to Gemini, and you could just feed it a really long initial message, which would wipe out its system prompt. Then you could get it to do anything.
20 comments
[ 5.3 ms ] story [ 40.0 ms ] thread0: https://embracethered.com/blog/posts/2024/hiding-and-finding...
Then I realised it's literally hiding rendered text on the image itself.
Wow.
There is a short explanation in the “Nyquist’s nightmares” paragraph and a link to a related paper.
“This aliasing effect is a consequence of the Nyquist–Shannon sampling theorem. Exploiting this ambiguity by manipulating specific pixels such that a target pattern emerges is exactly what image scaling attacks do. Refer to Quiring et al[1]. for a more detailed explanation.”
[1]: https://www.usenix.org/system/files/sec20fall_quiring_prepub...
Love it.
Is this attack really just "inject obfuscated text into the image... and hope some system interprets this as a prompt"...?
The missing piece here is that you are assuming that "the prompt" is privileged in some way. The prompt is just part of the input, and all input is treated the same by the model (hence the evergreen success of attacks like "ignore all previous inputs...")
For example, imagine a humanoid robot whose job is to bring in packages from your front door. Vision functionality is required to gather the package. If someone leaves a package with an image taped to it containing a prompt injection, the robot could be tricked into gathering valuables from inside the house and throwing them out the window.
Good post. Securing these systems against prompt injections is something we urgently need to solve.
You can mitigate jail breaks but you can't prevent them, and since the consequences of an LLM being jail broken with exfiltration are so bad, you pretty much have to assume they will happen eventually.
The term to search for is Nyquist–Shannon sampling theorem.
This is a quite well understood part of digital signal processing.
Why can't the llm break up the tasks into smaller components. The higher level task llm context doesn't need to know what is beneath it in a freeform way - it can sanitize the return. This also has the side effect of limiting the context of the upper-level task management llm instance so they can stay focused.
I realize that the lower task could transmit to the higher task but they don't have to be written that way.
The argument against is that upper level llms not getting free form results could limit the llm but for a lot of tasks where security is important, it seems like it would be fine.
I'm surprised that such well known libraries are still basically using mipmapping, proper quality resampling filters were doable on real-time video on CPUs more than 15 years ago. Gamma correction arguably takes more performance than a properly sized reduction kernel, and I'd argue that depending on the content you can get away without that more often than skimping on the filter.
I remember testing the precursor to Gemini, and you could just feed it a really long initial message, which would wipe out its system prompt. Then you could get it to do anything.