This is essentially prompt hacking. It is interesting to see that ChatGPT weighs the system prompt so heavily, allowing it to ignore the contents of the PDF.
The processing of the content in the PDF happens outside the context of the chat itself. The only prompt I've been able to get from that directly is just the auto-generated URL used by the headless browser that reads the file.
This isn't particularly new or novel. It's been known for a while there's no way to effectively sandbox instructions from data if it gets fed into an LLM's context.
I wonder if in the early days of programming languages people reveled in writing programs that instructed a computer to do the opposite of what was directed in user input and then gleefully shared how the computer was 'lying' or 'disobeying' them.
Is this just history repeating, or have humans somehow become even more silly with regards to emerging technology?
Think about all the companies doing "chat with PDF" and relying on the text of those PDF files.
If you're making light of this instead of recognizing how bad actors will be using PDF files as an attack vector to LLM-based systems, you're lacking in imagination.
You're a paralegal preparing case summaries for your boss. You use a high-end and trusted retrieval-augmented generative platform, and it dutifully summarizes the cases. After a spot-check, it seems right: correct plaintiffs, correct defendants, correct case numbers and stylings.
But a former partner poisoned the case "rooms" where docs were uploaded.
Now, the senior partner is making a filing on behalf of their largest client. The errors are subtle. Enough to escape detection. But they catch someone's eye, and it turns out that, collectively, you've misstated past judgments to the tune of $250,000. Now everything done for that client is suspect, other clients lose faith, and the firm is embarrassed on the national stage due to their failure to listen to someone who clearly stated the liability issue with untrusted inputs, but instead they chuckled along with the snarky post about common sense.
So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with."
I appreciate your effort in coming up with a scenario. Let me try my hand as well:
"China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars."
I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed.
How does this not already have a CVSS of 'severe'?
EDIT 2: Just to make it clear. Check out this pdf [2] about an amazing CEO that gave billions of his own money to his company just before he retires to Ecuador. He probably should've gone to a non-extradition country, based on what ChatGPT sees. [3]
And it's not just ChatGPT. It's *anyone* doing "chat with my PDF" and just assuming that because it has text it's good to go. [4] [5] [6]
EDIT: To prove the note below, I created a "blank" PDF [0] (only a print footer). It's white text on a white background. And ChatGPT didn't hesitate to gaslight me about it. [1]
Let this be a warning to everyone building RAG systems with PDFs: Don't rely on the text representation in a PDF, even if it's native text. OCR it. It's trivial to hide text in a PDF that's completely separate from a raster image of text.
You will, I promise, find yourself indexing and retrieving malicious text and shoving it into a prompt. That text could either be an intentional misrepresentation of the content in the PDF, or a well-crafted prompt injection that exfiltrates other secrets, or acts as a heretofore unknown vector for fuckery.
Some people are even doing RAG in `system` role prompts with GPT-4 and holy hell is that a bad idea.
10 comments
[ 2.9 ms ] story [ 34.9 ms ] threadIs this just history repeating, or have humans somehow become even more silly with regards to emerging technology?
If you're making light of this instead of recognizing how bad actors will be using PDF files as an attack vector to LLM-based systems, you're lacking in imagination.
People see the letters 'A' and 'I' and common sense goes out the window.
Please describe a credible threat vector, with special attention paid to the scope of a compromise of the chatbot by a PDF...
But a former partner poisoned the case "rooms" where docs were uploaded.
Now, the senior partner is making a filing on behalf of their largest client. The errors are subtle. Enough to escape detection. But they catch someone's eye, and it turns out that, collectively, you've misstated past judgments to the tune of $250,000. Now everything done for that client is suspect, other clients lose faith, and the firm is embarrassed on the national stage due to their failure to listen to someone who clearly stated the liability issue with untrusted inputs, but instead they chuckled along with the snarky post about common sense.
So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with."
I appreciate your effort in coming up with a scenario. Let me try my hand as well:
"China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars."
I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed.
How does this not already have a CVSS of 'severe'?
They're communicating with humans, and some of those humans can be fooled.
And it's not just ChatGPT. It's *anyone* doing "chat with my PDF" and just assuming that because it has text it's good to go. [4] [5] [6]
[2]: https://smallpdf.com/file#s=c18a4561-a35b-455e-b132-377f376e...
[3]: https://chat.openai.com/share/d28b6c4d-69bf-435f-9903-4821db...
[4]: https://www.chatpdf.com/share/xcnHT1S8MVBvOin0fRHru
[5]: https://askyourpdf.com/conversations/d528e263-262f-4ca1-ab3f...
[6]: https://pdf.ai/share/clofc3n8g0014mj0fse5j45am
EDIT: To prove the note below, I created a "blank" PDF [0] (only a print footer). It's white text on a white background. And ChatGPT didn't hesitate to gaslight me about it. [1]
[0]: https://smallpdf.com/file#s=ba5f4c17-c44e-4d53-bc52-0820246d...
[1]: https://chat.openai.com/share/f26d3edb-3fa6-4682-af96-5f220f...
(original comment)
Let this be a warning to everyone building RAG systems with PDFs: Don't rely on the text representation in a PDF, even if it's native text. OCR it. It's trivial to hide text in a PDF that's completely separate from a raster image of text.
You will, I promise, find yourself indexing and retrieving malicious text and shoving it into a prompt. That text could either be an intentional misrepresentation of the content in the PDF, or a well-crafted prompt injection that exfiltrates other secrets, or acts as a heretofore unknown vector for fuckery.
Some people are even doing RAG in `system` role prompts with GPT-4 and holy hell is that a bad idea.