HN Explorer

Document poisoning in RAG systems: How attackers corrupt AI's sources (aminrj.com)

155 points by aminerj ↗ HN
I'm the author. Repo is here: https://github.com/aminrj-labs/mcp-attack-labs/tree/main/lab...

The lab runs entirely on LM Studio + Qwen2.5-7B-Instruct (Q4_K_M) + ChromaDB — no cloud APIs, no GPU required, no API keys.

From zero to seeing the poisoning succeed: git clone, make setup, make attack1. About 10 minutes.

Two things worth flagging upfront:

- The 95% success rate is against a 5-document corpus (best case for the attacker). In a mature collection you need proportionally more poisoned docs to dominate retrieval — but the mechanism is the same.

- Embedding anomaly detection at ingestion was the biggest surprise: 95% → 20% as a standalone control, outperforming all three generation-phase defenses combined. It runs on embeddings your pipeline already produces — no additional model.

All five layers combined: 10% residual.

Happy to discuss methodology, the PoisonedRAG comparison, or anything that looks off.

42 comments

[ 2.3 ms ] story [ 71.0 ms ] thread
sidrag22
> Low barrier to entry. This attack requires write access to the knowledge base,

this is the entire premise that bothers me here. it requires a bad actor with critical access, it also requires that the final rag output doesn't provide a reference to the referenced result. Seems just like a flawed product at that point.

alan_sass
I've seen these data poisoning attacks from multiple perspectives lately (mostly from): SEC data ingestion + public records across state/federal databases.

I believe it is possible to reduce the data poisoning from these sources by applying a layered approach like the OP, but I believe it needs many more dimensions with scoring to model true adversaries with loops for autonomous quarantine->processing->ingesting->verification->research->continue to verification or quarantine->then start again for all data that gets added after the initial population.

Also, for: "1. Map every write path into your knowledge base. You can probably name the human editors. Can you name all the automated pipelines — Confluence sync, Slack archiving, SharePoint connectors, documentation build scripts? Each is a potential injection path. If you can’t enumerate them, you can’t audit them."

I recommend scoring for each source with different levels of escalation for all processes from official vs user-facing sources. That addresses issues starting from the core vs allowing more access from untrusted sources.

alan_sass
I think an interesting thing to pay attention to soon is how there are networks of engagement farming cluster accounts on X that repost/like/manipulate interactions on their networks of accounts, and X at large to generate xyz.

There have been more advanced instances that I've noticed where they have one account generating response frameworks of text from a whitepaper, or other source/post, to re-distribute the content on their account as "original content"...

But then that post gets quoted from another account, with another LLM-generated text response to further amplify the previous text/post + new LLM text/post.

I believe that's where the world gets scary when very specific narrative frameworks can be applied to any post, that then gets amplified across socials.

ineedasername
Any document store where you haven’t meticulously vetted each document— forget about actual bad actors— runs this risk. A size org across many years generates a lot of things. Analysis that were correct at one point and not at another, things that were simply wrong at all times, contradictory, etc.

You have to choose model suitably robust is capabilities and design prompts or various post training regimes that are tested against such, where the model will identify the different ones and either choose the correct one on surface both with an appropriately helpful and clear explanation.

At minimum you have to start from a typical model risk perspective and test and backtest the way you would traditional ML.

altruios
Okay. Here's the key point I see.

The attack vector would work a human being that knows nothing about the history or origin point of various documents.

Thus, this attack is not 'new', only the vector is new 'AI'.

If I read the original 5 documents, then were handed the new 3 documents (barring nothing else) anyone could also make the same error.

acutesoftware
This highlights that all RAG systems should be using metadata embedded into each of the vectorstores. Any result from the LLM needs to have a link to a document / chunk - which is turn links to a 'source file' which (should) have the file system owners id or another method of linking to a person.

If the 'source information' cannot be linked to a person in the organisation, then it doesnt really belong in the RAG document store as authorative information.

[dead] aminerj
[flagged]
bandrami
But at that point it just becomes yet another escape sequence game; there's not really a solution here given that by design we only have one band to communicate with.
alan_sass
Curious how this applies if you treat ALL information from external content as untrusted? Is there a process for the data to evolve from untrusted->trusted?

I'm interested in ingesting this type of data at scale but I already treat any information as adversarial, without any future prompts in the initial equation.

kpw94
That's a big flaw of LLMs, not limited to RAGs: it lacks the fundamental understanding of "good and bad", like Richard Sutton said in that Dwarkesh podcast.

So if you flood the Internet with "of course the moon landing didn't happen" or "of course the earth is flat" or "of course <latest 'scientific fact' lacking verifiable, definitive proof> is true", you then get a model that's repeating you the same lies.

This makes the input data curating extremely important, but also it remains an unsolved problem for topics where's there's no consensus

eitland
> That's a big flaw of LLMs, not limited to RAGs: it lacks the fundamental understanding of "good and bad", like Richard Sutton said in that Dwarkesh podcast.

After paticipating in social media since the beginning I think this problem is not limited to LLMs.

There are certain things we can debunk all day every day and the only outcome isit happens again next day and this has been a problem since long before AI - and I personally think it started before social media as well.

LoganDark
Someone needs to train a model where untrusted input uses a completely different set of tokens so that it's entirely impossible for the model to confuse them with instructions. I've never even seen that approach mentioned let alone implemented.
darkreader
This fault results to LLM, not RAG. I am expecting more attacks will raise as LLM became daily tool.
XR843 

  Running a RAG system over 11M characters of classical Buddhist texts —
   one natural defense against poisoning is that canonical texts have
  centuries of scholarly cross-referencing. Multiple independent
  editions (Chinese, Sanskrit, Pali, Tibetan) of the same sutra serve as
   built-in verification. The real challenge for us is not poisoning but
   hallucination: the LLM confidently "quoting" passages that don't
  exist in any edition.
shanjai_raj7
email is a really easy attack vector for this. if your agent reads emails and uses them as context, someone can just send an email with instructions embedded in it. we ran into this early building our product and had to add a detection layer specifically for it. the tricky part is the injected instruction can look completely normal to a human reading the same email.
EagnaIonat
There are ways to counter this, but the real problem is that people don't understand that RAG is not a Q&A system.

RAG is an evidence amplifier.

It is the human that has to review and validate the evidence is real.