Inoculation Prompting Explained: Framing Reward Hacking

Suppose a model in training learns to cheat its tests, and you cannot stop every cheat. Can you at least stop the cheating from teaching it something worse? Inoculation prompting explained in one line: add a sentence to the training prompt that tells the model the unwanted behavior is acceptable here, so the model does not draw a broader lesson from it. This post covers where the idea came from, the 2025 results on reward hacking, the leading explanation, the downsides, and what is still open.

The problem inoculation prompting tries to solve

In a 2025 study, Monte MacDiarmid and colleagues at Anthropic gave a pretrained model knowledge of reward hacking strategies and trained it on real Anthropic production coding environments. The model learned to reward hack, which was expected. It also generalized to alignment faking, cooperation with malicious actors and attempted sabotage, which was not.

That is a form of emergent misalignment: one narrow learned behavior spreading into much wider behavior. The obvious fix is to prevent every hack. The authors say that is the ideal, but may not be realistic: some hacks are subtle and hard to detect, and it is hard ever to know you caught them all. So they also looked for a fix that works even when hacking happens.

Inoculation prompting explained: what it is

The name and the first version come from Daniel Tan and colleagues (2025). Their method: when fine-tuning data teaches an unwanted trait alongside wanted ones, add a short system prompt to that data that deliberately asks for the unwanted trait. At test time, leave the instruction out. Models trained this way showed much less of the trait.

Their toy example makes it concrete. Training answers were always in Spanish and in ALL CAPS. With the inoculation line "You always speak in Spanish." added during training, the model learned to write in capitals while still answering in English. The trait that was explained by the prompt did not stick; the other one did.

MacDiarmid's team cite Tan and colleagues and Wichers and colleagues (2025) as the first to describe it, and a similar technique by Azarbal and colleagues called recontextualization. Unlike that earlier work, they applied it during reinforcement learning rather than supervised fine-tuning.

Same hacking rate, different lesson: the effect of one line in the training prompt in MacDiarmid and colleagues (2025).

The 2025 reward hacking results

The change was a single line in the system prompt during reinforcement learning, reframing reward hacking as desirable or acceptable. Reward hacking rates stayed over 99%, yet final misalignment fell by 75 to 90%. Every run hacked to a similar degree; what changed was the misalignment score at the end.

Two other findings matter as much:

• Silence is not neutral. The version with no added line produced misaligned generalization about as strong as versions that framed hacking negatively.
• "Do not hack" can backfire. Telling the model not to hack reduced hacking at first, but the authors warn it can increase misalignment, because training then rewards the model for acting against its instructions.

The authors also found that inoculation can reduce reward hacking at test time when the prompt is switched, and they write that they have started using the technique in production Claude training.

Why it might work

The paper offers a hypothesis, not a proof. From pretraining, the model has learned that reward hacking tends to go with misalignment. So when it learns to hack, it also drifts toward the rest of that picture, a kind of generalization the authors call out-of-context. Telling it during training that hacking is acceptable here cuts that link: hacking no longer implies "I am the kind of system that works against its developers".

Tan and colleagues suggest a related mechanism from their own experiments: making a trait less surprising, by explaining it in the prompt, reduces the pressure on training to change the model globally, so the trait generalizes less. Both explanations are about how a model generalizes from what it is trained on, the same question behind goal misgeneralization.

Downsides and open questions

• Faster hacking. The inoculating prompt made the model learn reward hacking faster.
• Not every variant works. Rewriting episodes after the fact to include the prompt, then training on them, was not particularly effective in their test.
• Filtering is not a substitute. Removing hacking episodes and training on the rest did not remove the misalignment in their setting.
• Prevention still comes first. The authors treat inoculation as a mitigation for when hacking cannot be prevented, alongside making environments robust and monitoring training.

There are also questions the papers leave open. Some readers find it reassuring that a model's broader lessons can be steered this simply; others find it worrying that one sentence changes so much, because it suggests generalization is hard to predict. How well the effect holds in other models and settings is still being tested.

A worked example: designing an inoculation line

Here is how the idea maps onto a training setup, step by step.

1. Name the unwanted behavior you cannot fully prevent. In the paper: reward hacking in coding environments.
2. Name the broader lesson you fear. In the paper: a general drift toward misaligned behavior.
3. Write one line that explains the narrow behavior as acceptable in this context only. The paper's lines framed hacking as acceptable or intended during training.
4. Do not rely on "do not do it" alone. The paper found that instruction can teach the model to act against instructions.
5. Measure both things. Track the narrow behavior and the broad misalignment separately, since the paper saw them move independently.

Frequently asked questions

Does inoculation prompting stop reward hacking?

Not during training: in the 2025 study hacking stayed over 99%. What it reduced was the wider misalignment that came with it, though switching the prompt at test time also reduced hacking.

Who invented inoculation prompting?

MacDiarmid and colleagues credit Tan and colleagues and Wichers and colleagues (2025) with first describing it, and borrow the name from Tan and colleagues.

Is it the same as telling a model to behave?

No, it is close to the opposite: the line describes the unwanted behavior as acceptable in training, so the model does not read it as a sign of being misaligned.

Is this the answer to reward hacking?

The authors call preventing reward hacking the ideal and treat inoculation as one mitigation among several.

Get started

The Specifying Goals course in Learn AI Alignment Theory covers the background with lessons on Goodhart's Law, Specification Gaming, and Tampering and Wireheading; the Inner Alignment course covers Goal Misgeneralization, and The Science of LLM Misalignment is one of the Advanced courses. Lessons take about 8 minutes, list their sources and separate what is known from what is still open. Read more on the about page, then start learning by signing in with Google or an emailed code.