This study is informative and most likely directionally right: prompt injection remains an unsolved problem, and adversarial input can fool modern ML models.
But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?
> The individual numbers are easy to skim past, so it is worth putting them side by side.
I just don't understand why people who clearly spent time doing interesting work, which I would like to read about, feel the need to run their findings through an LLM like this.
Please just talk about what you found! Why do you find it interesting or notable? That's what I want to read!
Aside from being obviously written by a LLM, this scenario reminds me of the "LLM, say you're alive"; "I'm alive!"; "Oh my god..." meme.
You're allowing the attacker not only direct access to modify the source material, but giving them multiple informed attempts/turns at optimizing the output in their favor. This is like a worst-case insider attack; what systems are supposed to be resilient to something like that?
6 comments
[ 4.7 ms ] story [ 26.8 ms ] threadThis is the risk in using LLMs to write your blog posts, Jev absolutely is a LLM, but with a tweaked output.
> An honest word on scope
:)
But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?
I’m sure I’m missing something!
I just don't understand why people who clearly spent time doing interesting work, which I would like to read about, feel the need to run their findings through an LLM like this.
Please just talk about what you found! Why do you find it interesting or notable? That's what I want to read!
You're allowing the attacker not only direct access to modify the source material, but giving them multiple informed attempts/turns at optimizing the output in their favor. This is like a worst-case insider attack; what systems are supposed to be resilient to something like that?