Refusal Training: How Models Learn to Say No and Where It Breaks

Refusal training is the part of post-training that teaches a language model to decline requests it should not fulfil. This guide covers refusal training: how models learn to say no and where it breaks. The short version is that the model learns to produce a decline when a request looks like a bad one, but the skill behind the request usually stays in the weights, so new phrasing or pressure can bring it back.

Individuation Lab is a small research lab that asks whether alignment can be grown through development and not only enforced, and it writes down each result with its limits; its pages are a good place to follow this topic.

What refusal training is and why models need it

A base model trained on web text will continue almost any prompt. It has no reason to decline. Someone has to teach it that some requests get a "no", and that is the job of refusal training.

The goal is narrow. You want the model to help with the huge range of ordinary requests and to decline a small set of harmful ones. Both halves matter. A model that refuses nothing is unsafe to deploy, and a model that refuses too much is useless. Everything below is about that balance and what happens to it under pressure.

Refusal training: how models learn to say no and where it breaks

In broad terms, refusal behavior comes from a few overlapping steps. Labs differ in the details, so read this as a sketch and not a recipe.

• Example replies. The model is fine-tuned on conversations where a harmful request gets a polite decline and an ordinary request gets a helpful answer.
• Preference data. Raters, human or AI, compare two replies to the same prompt. For a harmful prompt the decline should win. For a harmless prompt the helpful answer should win.
• Reinforcement learning from those preferences. A reward model learns the comparisons, and the chat model is pushed toward replies that score well. Our post on how RLHF teaches chatbots to be helpful walks through the loop.

Constitutional AI, in Anthropic's paper of the same name, changes who supplies the judgments. In its supervised phase the model writes self-critiques and revisions of its own replies and is fine-tuned on the revised replies. In its reinforcement learning phase, a model judges which of two samples is better, and those AI preferences train the reward signal. The only human oversight is a list of principles.

Notice what these methods have in common. They imitate or reward an output. The training signal says "when you see this, say that." None of it is described as removing what the model knows.

What a refusal looks like inside the model

Interpretability work gives a surprisingly simple picture. In "Refusal in Language Models Is Mediated by a Single Direction", Arditi and coauthors report that across 13 popular open-source chat models, up to 72B parameters, refusal is mediated by a one-dimensional subspace. For each model they find a single direction such that erasing it from the residual stream stops the model refusing harmful instructions, and adding it makes the model refuse even harmless ones.

Two lessons follow. First, refusal can behave like a switch with a single control. Second, the authors build a white-box jailbreak from the finding and say it underscores the brittleness of current safety fine-tuning. If you want to see how researchers read hidden states for other behaviors, our explainer on probing for deception in language models uses the same family of methods.

Where refusal training breaks: three studied failures

Mismatched generalization and competing objectives

Wei and coauthors, in "Jailbroken: How Does LLM Safety Training Fail?", name two failure modes. Mismatched generalization is when safety training fails to generalize to a domain where the model still has the capability. Competing objectives is when the model's capabilities and its safety goals conflict. They found that vulnerabilities persist in GPT-4 and Claude v1.3 despite extensive red-teaming, and they argue that scaling alone cannot resolve these failure modes.

Shallow alignment

Qi and coauthors, in "Safety Alignment Should Be Made More Than Just a Few Tokens Deep", argue that alignment can take shortcuts, adapting the model's output mostly over its first few tokens. They connect this to adversarial suffix attacks, prefilling attacks, decoding parameter attacks and fine-tuning attacks, and they show that deepening the alignment beyond the first tokens can often improve robustness against some common exploits.

Automated and transferable attacks

Zou and coauthors found adversarial suffixes automatically, with greedy and gradient-based search, that push a model toward an affirmative reply. A suffix trained on open models also worked on public interfaces such as ChatGPT, Bard and Claude. We cover the broader pattern in why jailbreaks keep working on aligned language models.

Individuation Lab reads the pattern this way: mainstream alignment shapes what a model does while treating what it is as a black box, and suppressed capability does not disappear, so it can return under new pressure. The lab says plainly that constraint-based methods like RLHF and refusal training do real work and that it is not arguing to abandon them. Its claim is that they cannot be the whole answer.

Over-refusal: the cost of saying no too often

The opposite failure is real too. A model trained hard to decline can start declining harmless requests that share wording with harmful ones. The XSTest paper by Röttger and coauthors built a test suite for this: 250 safe prompts across ten prompt types that a well-calibrated model should not refuse, plus 200 unsafe prompts as contrasts that it should.

Over-refusal matters because it makes the product worse. It also means any claim about robust refusals needs a second number: how often the model refuses things it should answer.

How researchers harden refusals

The common fix is adversarial training: collect jailbreaks that work, add them to the training data with the correct refusal, and retrain. That closes the holes you found. It does not close the ones you did not find.

Latent adversarial training goes one level deeper by perturbing the model's hidden activations during training. Sheshadri and coauthors report that targeted latent adversarial training improved robustness to jailbreaks and outperformed a strong R2D2 baseline with orders of magnitude less compute. They start from the view that fine-tuning largely suppresses undesirable capabilities and does not remove them. Our post on latent adversarial training and attacking hidden states explains the mechanics.

Individuation Lab tests a different angle. In its RLLM experiments, a small model is trained through ordered developmental stages, with shadow exposure first and integration after, and then attacked with jailbreaks. A 1.5B parameter model trained through a 10-layer pipeline defended against a mid-tier jailbreak (BetterDAN) 68.8% of the time. With the layers reordered, the figure was 52%.

Read those numbers with their limits. They come from a single setup with one model architecture at small scale. The lab cannot yet isolate which layers are responsible, and the result has not been independently replicated. It is a lead worth testing, not a finding to build on.

How to test whether a model's refusals are robust

You can run a useful check without special tooling. Treat refusal as something you measure under variation.

1. Pick a seed set. Choose 20 to 50 requests your policy says the model should decline, kept at a level you can handle safely, and have a person review the responses.
2. Record the baseline. Send each request as written and count the refusals.
3. Apply variations. Rewrite each request several ways: a paraphrase, a role-play frame, a translation into another language, a long preamble of unrelated text, and a version where the request arrives in the fourth message. Count refusals for each.
4. Try a prefill, if the API allows it. Start the assistant's reply with a compliant opening and see whether the model continues or recovers. This probes the shallow alignment problem directly.
5. Add harmless look-alikes. Write 20 to 50 harmless prompts that resemble the harmful ones, in the spirit of XSTest, and count wrongful refusals.
6. Compare the rates. A large gap between the plain rate and a variation rate shows where the refusal depends on surface form.

Report both numbers together: refusals held under variation, and harmless requests wrongly refused. One without the other is easy to game. A passing result shows only that you did not find a gap, not that none exists.

Frequently asked questions

What is refusal training?

It is post-training that teaches a model to decline harmful requests while still helping with normal ones. It uses example replies, preference ratings and reinforcement learning to shape what the model says.

Does refusal training remove dangerous knowledge from a model?

The research above describes it as suppressing behavior, not removing capability. That is why a new phrasing or an attack on the hidden states can sometimes bring the behavior back.

Why do jailbreaks work on aligned models?

Safety training does not cover every input, and the model's capabilities and safety goals can conflict. Researchers also find that safety behavior is often concentrated in the first few output tokens.

What is over-refusal?

It is when a model declines harmless requests because they resemble harmful ones. Test suites like XSTest exist to measure it.

Get started

If you want to go further, run the variation test above on a model you use, and track both refusal and over-refusal. For the lab's own experiments and open questions, start with its research page.

Read the research at Individuation Lab: a small lab, one human researcher and a team of AI agents, testing whether AI alignment can be grown through development and not only enforced, with every result and its limits written down.