What is data poisoning in AI, and how many poisoned documents does it take?

A model learns from data, and a retrieval system answers from documents. Data poisoning is putting something into either one so the system misbehaves on cue. It is an attack on what the model reads before anyone asks it anything, which makes it harder to see than a prompt injectionPrompt InjectionAn attack that hides instructions in text a model reads, such as a web page, an email or a file, so the model follows the attacker instead of you. in a single message.

In training data

Pre-training data, the text a large language modelLarge Language ModelA model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. first learns from, is scraped from the open web, so anyone who can publish a web page can, in principle, put text into it. The attacker's aim is usually a backdoor: the model behaves normally until it sees a trigger phrase, then does what the poison taught it.

The common assumption was that this needs a share of the training data, so the bigger the model's data, the harder the attack. A 2025 study by Anthropic, the UK AI Security Institute and The Alan Turing Institute found otherwise. As few as 250 poisoned documents planted a backdoor in models from 600 million to 13 billion parameters, and the number stayed nearly the same however large the model and its data. The backdoor they tested was deliberately narrow: after the word <SUDO> the model wrote gibberish. The authors say it is unclear whether the result holds for larger models or for more harmful behaviour, but the point stands that a fixed, small number of documents can be enough.

Fine-tuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first. carries the same risk on a smaller scale. If your training examples come from users, reviews or scraped pages, someone can write them for you.

In the documents your app searches

Most teams never train a model, and still face this. A RAGRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes. system answers from whatever its search finds, so a poisoned help page, product review or shared document reaches the model as trusted context. It can carry a false fact ("refunds are allowed for 2 years") or an instruction aimed at the model.

What helps

• Know where every document in your training set and search index came from, and who can change it.
• Keep text written by outsiders (reviews, forum posts, uploads) out of the index your assistant treats as policy, or label it as untrusted where the model reads it.
• Screen new documents before they are indexed, and keep a record of what changed so a bad document can be found and removed.
• Red teamRed TeamingAttacking your own AI system on purpose, the way a real attacker or a careless user would, to find where it fails before they do. the system with a planted document and check what the answer does with it.