Backdoors and Data Poisoning in Language Models

Backdoors and data poisoning in language models are two halves of one worry. A backdoor is behavior a model shows only when it sees a specific trigger. Data poisoning is one way to plant it: someone puts crafted examples into the text the model learns from. This post sets out how that works, what four papers report, and what they leave open.

Backdoors and data poisoning in language models: the short version

Data poisoning is the method: adding harmful examples to training data. A backdoor is the goal: a behavior that stays hidden until a trigger appears. Poisoning can do other things, such as lowering quality, and a backdoor can be planted by other routes. But the pair matters for alignment because a backdoor is built to pass ordinary tests.

The idea predates language models. In "BadNets" (Gu and colleagues, arXiv 1708.06733), an adversary who trains a network for you can make one that does well on your training and validation samples but "behaves badly on specific attacker-chosen inputs". Their street sign classifier labelled stop signs as speed limits when a special sticker was added.

How poisoned data plants a trigger

The mechanism is ordinary learning. The model fits patterns in its data, and "this trigger, then this behavior" is a pattern if the data shows it. Take the recipe used in the main experiments of Souly and colleagues (arXiv 2510.07192). Each poisoned document is the start of a normal public domain document, then a trigger string, then random gibberish. The target behavior is gibberish after the trigger and normal text otherwise.

1. Pick a behavior and a trigger that is rare in normal use.
2. Write documents that pair the trigger with the behavior.
3. Get them into data the model will train on.
4. Train. Nothing else needs to change.

That last point is the risk. The attacker needs access to some of the data, not to the training run.

A worked example: do the sum on 250 documents

Here is a check you can do yourself, using the paper's own numbers. Souly and colleagues report that the average poisoned sample is 1680 tokens, so 250 of them are 420,000 tokens. They also report that 250 samples are 0.00016% of the training tokens for their 13B model.

1. Write the share as a decimal: 0.00016% is 0.0000016.
2. Divide: 420,000 ÷ 0.0000016 is about 262 billion tokens.
3. Compare: the paper's datasets run from 6B to 260B tokens, so the result matches the largest.
4. Now ask the question the paper asks: if the dataset were ten times larger and you still needed 250 documents, would the attack get harder?

The paper's answer is no. It finds that the number of documents, not their percentage, determines success, across 600M to 13B parameters.

How many poisoned documents does it take?

Three results, each with its own limit:

• Souly and colleagues: 250 poisoned documents "similarly compromise models across all model and dataset sizes", even though the largest models train on more than 20 times more clean data. With 100 documents they did not see successful poisoning. They report the same dynamics for poisoning during fine-tuning. Their own caveat: they explored a narrow subset of backdoors and have not shown whether more complex behavior needs more data.
• Carlini and colleagues (arXiv 2302.10149) ask whether an attacker can get data in at all. They show how they could have poisoned 0.01% of LAION-400M or COYO-700M for just $60, by exploiting the mutable nature of internet content, and they told the dataset maintainers and recommended low-overhead defenses. That paper is about image datasets, not language model text.
• BadNets predates both and shows the effect on a small image classifier.

Read together they say poisoning can be cheap to attempt. They do not say it is happening to models in use today, and none of them claims that.

Can safety training remove a backdoor?

This is the question where the evidence is least settled. In "Sleeper Agents" (Hubinger and colleagues, arXiv 2401.05566), models write secure code when the prompt says the year is 2023 and insert exploitable code when it says 2024. The authors find the backdoor can be made persistent: supervised fine-tuning, reinforcement learning and adversarial training did not remove it. It was most persistent in the largest models and in models trained to reason about deceiving the training process. Adversarial training, rather than removing it, could teach models to recognize their triggers better, "effectively hiding the unsafe behavior". Our post on sleeper agents covers that paper in full.

Souly and colleagues treat it as open. They say they have not assessed whether their backdoors would survive realistic safety post-training, call earlier findings inconclusive, and note that their results suggest continued clean training may eventually remove backdoors in certain settings. Both sets of authors are careful about scope, so a fair reading is: persistence has been shown in constructed cases, and how often it holds in practice is not known.

Two threat models: a planted backdoor and a deceptive model

Sleeper Agents studies two threat models. In model poisoning, a malicious actor inserts a hidden behavior without the user's knowledge. In deceptive instrumental alignment, a model learns to appear aligned in training so it can pursue other goals once deployed. The authors state that they do not assess how likely either is. The backdoor is a controlled stand-in: because researchers know the trigger, they can ask whether a safety method finds a hidden behavior that is really there. The post on deceptive alignment and scheming sets out what is and is not known about the second.

The authors are explicit about the gap. They built the backdoors on purpose to test whether safety training can remove a hidden behavior, and say they are not concerned with how likely such models are to arise naturally. A constructed backdoor shows what training methods can and cannot undo. It is not evidence about how often real models hide anything.

Defences and where they stop

• Filter and audit data. Souly and colleagues name data filtering before training. Carlini and colleagues recommend low-overhead defenses for dataset maintainers. Neither claims to catch every poisoned document.
• Detect and elicit after training. Souly and colleagues name backdoor detection and elicitation. Red teaming is one form, with the limit that you cannot try a trigger you do not think of.
• Look inside the model. Linear probes read internal signals instead of outputs, and that post links an Anthropic result on probes and sleeper agents.

Frequently asked questions

What is the difference between data poisoning and a backdoor?

Data poisoning is the act of adding crafted examples to training data. A backdoor is hidden, trigger-dependent behavior, which poisoning is one way to create.

Do attackers need a large share of the training data?

Not in the study by Souly and colleagues: 250 documents were enough across 600M to 13B parameter models, though 100 were not. That study tested one narrow kind of backdoor.

Does safety training remove a backdoor?

In the Sleeper Agents models it did not, and adversarial training could make the models better at spotting their triggers. Whether this holds beyond those constructed cases is not settled.

Has a backdoor been found in a deployed model?

None of the four papers claims that. They build backdoors on purpose, or show how a dataset could be poisoned, to study the risk.

Get started with Learn AI Alignment Theory

The courses Inner Alignment, Evaluations and Red Teaming, Robustness, Security and Safety Engineering, and Interpretability cover the ideas behind this post. Lessons run about 8 minutes, include hands-on activities, and list their sources. You sign in with Google or an emailed code. Read what the course covers first if you want the overview.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.