This post covers a coordinated disclosure with Microsoft (MSRC) regarding a vulnerability class that allows attacker-controlled instructions in an attached document to hijack Copilot for Word.
It manipulates the AI to alter the output text (e.g., halving financial figures) and append the attack prompt into the new document concealed as white text.
Because the downstream document now carries the payload, it acts similarly to an AI worm across normal user workflows. Microsoft deployed multiple fixes over a 144-day coordination period, but the broader vulnerability class remains unmitigated and exploitable because it exploits fundamental limitations of current LLMs.
When attacker instructions are combined with legitimate information the model's context window, the tokens being inspected participate in the act of inspection, meaning current LLM architectures provide no reliable boundary between intention and interpretation.
> Malicious instructions hidden in an externally shared document could make Copilot alter drafted or edited documents in Word and propagate the attack to new documents.
It's not really important whether the text is visible, you can just put the instructions in the middle of a long document, or use a low contrast color, or a small font size, or any other of the myriad of ways to make text hard to read.
I may be naive here but can the hidden text not be flagged or outright removed before being passed to copilot? Why would there not be consideration for what a human user can see, especially if the hidden text was added by copilot in the first place?
LLMs should be viewed with the same terror as a reckless toddler who knows some bash syntax. Deeply embedding them into important and privileged systems will be the end of us.
Now I’m imagining the horrible 19th-century style psychological torture where they take some Victorian orphan—an infant tucked away with a tattered stuffed animal, confined to a cheap facsimile of a domestic home within the corridors of a dank sanitarium full of tuberculosis, leprosy, the clinically insane, and floors of hysterical women receiving lobotomies—and they raise that child from birth to do nothing else but write Bash scripts.
Some would call it “cruel”, and I do admit: Bash can be a frustrating language...but these are the unspeakable things we do in the name of progress.
I'm a programmer and a web-based AI user, but I don't want AI running on my local machine in any form. I've uninstalled Copilot and disabled AI in all local applications including the browser itself for exactly the reason described in this article. There's no way to protect your data from such an AI confusion attack by design. AI cannot discern your prompts versus text in file. The fact that an AI enabled word processor or email app could follow instructions embedded in a run-of-the-mill document or email is insane. Switching to Linux, BSD or another open source operating system is the only real solution to this problem.
There are many approaches today. Check out https://tritium.legal/blog/noroboto where we tricked frontier algorithms into reading different Unicode values from those presented by the fonts in the document.
No, make corporations actually liable/responsible for their harmful choices in pursuit of quarterly profits, and then they'll stop using the technology badly.
This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system
Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know what you're signing up for if you sell yourself to anthropic or openai
> people let AI agents have full unrestricted access to their system
Not everyone is like this. I've never ran Claude Code unsandboxed. I took the time to firejail it since day one. One of the first things I did was ask it to write a virtual machine orchestration tool so I could virtualize entire projects and their associated AI sessions. Mitigates supply chain attacks too.
I managed to rig up a firewall and it withstood a Fable code review. I'm now writing a custom network stack for it in Rust so I can firewall every single network connection.
Any data that is consulted in the course of deciding how to use the capabilities of a system are instructions (they are a different and less privileged level of instruction than the set of instructions which provide the capabilities, unless one of the capabilities of the system is updating those privileged instructions, in which case that separation disappears.)
with LLM architectures I think that’s true, and we don’t have anything looking particularly competitive for large scale use atm
I also don’t think this is about mixing, the LLM part of the problem doesn’t have determinism around the boundaries so they’re feel good at best, maybe making some cases a bit harder
trifecta is a forever problem with this architecture
I am surprised to not see more literature on how LLMs are layered to handle complex decision making including security.
I’m assuming most companies aren’t just routing user prompts/file uploads to a single LLM and returning the reply back hahah
Even file uploads - I would strip content and only support certain file types. The stripped text would be analyzed.
There wouldn’t be a concept of “hidden text” since it’s not going through a vision model. It’s just text. The threat is the same as any other prompt injection.
Hiding the text in the document would have no additional effect.
52 comments
[ 70.2 ms ] story [ 562 ms ] threadThis post covers a coordinated disclosure with Microsoft (MSRC) regarding a vulnerability class that allows attacker-controlled instructions in an attached document to hijack Copilot for Word.
It manipulates the AI to alter the output text (e.g., halving financial figures) and append the attack prompt into the new document concealed as white text.
Because the downstream document now carries the payload, it acts similarly to an AI worm across normal user workflows. Microsoft deployed multiple fixes over a 144-day coordination period, but the broader vulnerability class remains unmitigated and exploitable because it exploits fundamental limitations of current LLMs.
When attacker instructions are combined with legitimate information the model's context window, the tokens being inspected participate in the act of inspection, meaning current LLM architectures provide no reliable boundary between intention and interpretation.
Well, that sounds promising..
Oh no.
It is fun to see how all AI narratives are collapsing.
Purged I would have
All things Microsoft from my (controllable) world
Some would call it “cruel”, and I do admit: Bash can be a frustrating language...but these are the unspeakable things we do in the name of progress.
Paste any document in any LLM and you'll risk that, it's not something Microsoft specific.
There are many approaches today. Check out https://tritium.legal/blog/noroboto where we tricked frontier algorithms into reading different Unicode values from those presented by the fonts in the document.
Oh who am I kidding, ya'll asked for this reality. I will take great joy in the suffering from my AI-less soapbox.
Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.
Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know what you're signing up for if you sell yourself to anthropic or openai
Not everyone is like this. I've never ran Claude Code unsandboxed. I took the time to firejail it since day one. One of the first things I did was ask it to write a virtual machine orchestration tool so I could virtualize entire projects and their associated AI sessions. Mitigates supply chain attacks too.
I managed to rig up a firewall and it withstood a Fable code review. I'm now writing a custom network stack for it in Rust so I can firewall every single network connection.
I also don’t think this is about mixing, the LLM part of the problem doesn’t have determinism around the boundaries so they’re feel good at best, maybe making some cases a bit harder
trifecta is a forever problem with this architecture
I’m assuming most companies aren’t just routing user prompts/file uploads to a single LLM and returning the reply back hahah
Even file uploads - I would strip content and only support certain file types. The stripped text would be analyzed.
There wouldn’t be a concept of “hidden text” since it’s not going through a vision model. It’s just text. The threat is the same as any other prompt injection.
Hiding the text in the document would have no additional effect.