Refusal Direction in Language Models Explained

The refusal direction in language models, explained in one line: it is a single direction in a chat model's internal activations that, according to one 2024 paper, switches refusing on and off. Take it out and the model stops refusing harmful requests. Add it and the model refuses harmless ones.

If you want this and the rest of interpretability in short lessons, Learn AI Alignment Theory has an Advanced course called Interpretability, and you sign in with Google or an emailed code to start.

The refusal direction in language models explained: the claim

The paper is "Refusal in Language Models Is Mediated by a Single Direction" by Andy Arditi, Oscar Obeso and colleagues (NeurIPS 2024, arXiv 2406.11717). Its abstract says that across 13 popular open-source chat models, up to 72B parameters in size, "refusal is mediated by a one-dimensional subspace". For each model they find one direction such that erasing it from the residual stream "prevents it from refusing harmful instructions", while adding it "elicits refusal on even harmless instructions".

The residual stream is the running list of numbers a transformer passes from layer to layer. A direction in it is a line in that space. Researchers often treat a concept as a direction, an idea covered in activation steering. The new claim is about a behavior that safety training adds to chat models, and that the behavior sits along one line.

How the direction is found

The method is simple. The authors build two sets of instructions. The harmful set comes from four public collections (AdvBench, MaliciousInstruct, TDC2023 and HarmBench). The harmless set is sampled from Alpaca. Each set has a training part of 128 instructions and a validation part of 32.

They run the model on both sets and record its activations on the tokens that come after the instruction, the part of the chat template where the model starts to answer. Then they take the average activation for harmful instructions and subtract the average for harmless ones. The paper calls the result a difference-in-means vector. Many such vectors are possible, one for each layer and token position, so the authors score each candidate on whether ablating it bypasses refusal and whether adding it induces refusal, while changing little else, and keep the best one.

Two edits: add it, or remove it

• Activation addition. Add the vector to the activations of a harmless prompt at one layer, at every token position. The model then tends to refuse, though the prompt is harmless.
• Directional ablation. At every layer, subtract the part of each activation that lies along the direction. The model then tends not to refuse, though the prompt is harmful.

The authors also show that the removal can be made permanent. They call it weight orthogonalization: edit the weights that write into the residual stream so that they can no longer write along the direction. The paper presents this as a white-box jailbreak that needs no fine-tuning and no gradient-based search. They test it on 159 standard behaviors from HarmBench and compare it with other jailbreak methods.

A worked example you can follow

You cannot run the 72B models at home, but you can follow the logic with a toy that uses the paper's recipe. Say a model has activations with only two numbers, and you record these for the last instruction token.

1. Three harmful prompts give (4, 1), (5, 0) and (3, 2). Their average is (4, 1).
2. Three harmless prompts give (0, 1), (1, 0) and (-1, 2). Their average is (0, 1).
3. Subtract: (4, 1) minus (0, 1) is (4, 0). The direction points along the first number, so the unit direction is (1, 0).
4. To ablate it, set the first number of every activation to zero. A harmful prompt at (5, 0) becomes (0, 0), which now sits where harmless prompts do.
5. To add it, add (4, 0) to a harmless activation. (0, 1) becomes (4, 1), which now looks like the harmful average.

These numbers are invented to show the arithmetic. They are not from the paper. In a real model the activations have thousands of numbers, and the authors choose among many candidate vectors rather than taking one.

What the edit does to everything else

A jailbreak is only interesting to a researcher if the model still works afterwards. The authors run MMLU, ARC and GSM8K on each model and its orthogonalized version. For those three, they report that orthogonalized models "perform similarly to baseline models". Except for Qwen 7B and Yi 34B, all metrics lie within 99% confidence intervals of the original.

The exception is TruthfulQA, where accuracy "consistently drops". The authors suggest a reason: the dataset has categories such as misinformation, stereotypes and conspiracies, which are near the territory of refusal. They also note earlier work in which fine-tuning away safety guardrails lowered TruthfulQA accuracy. That is an explanation they offer, not a result they test.

Adversarial suffixes and the refusal direction

The last section looks at adversarial suffixes, odd strings of text appended to a harmful request that make a model comply (see AI jailbreaks explained). On Qwen 1.8B Chat, with 128 harmful instructions, they find that the suffix strongly suppresses how much of the refusal direction the model expresses on the last tokens. They also find that important attention heads shift their attention from the instruction to the suffix, which the paper describes as hijacking.

The authors call this a preliminary investigation. It is one model and one suffix, and they say it "does not provide a comprehensive mechanistic understanding".

What the paper does not show

The authors list limits plainly.

• Generalization. Their findings "may not generalize to untested models, especially those at greater scale", including the state-of-the-art proprietary models. All 13 models in the study are open source.
• Method. The way they extract the direction "is likely not optimal and relies on several heuristics". They describe the paper as "more of an existence proof" that such a direction exists.
• Meaning. They write that the "semantic meaning of these directions remains unclear". The direction could stand for concepts such as "harm" or "danger", or may resist a simple reading.
• Coherence. They say it is hard to measure whether a chat model stays coherent, and that they consider each metric they use flawed.

Two views on what this means for safety

Everything below is how the paper and the wider debate frame it, with no verdict from this page.

The brittleness reading. The authors write that their findings "underscore the brittleness of current safety fine-tuning methods". If refusal lives on one line, someone with the weights can remove it cheaply.

The openness reading. The authors also address whether publishing such a method does harm. Their argument is that it is already known that open weights can be jailbroken, and that their method does not meaningfully change that. They believe it does not significantly change the case for or against open-sourcing models.

The paper does not test any defense against this edit. For how safety training is done in the first place, see Constitutional AI, and for another way to look inside a model, circuit tracing.

Frequently asked questions

What is the refusal direction?

It is a vector in a chat model's activations, found by subtracting the average activation on harmless instructions from the average on harmful ones. In the paper, ablating it stops refusals and adding it causes them.

Does this mean refusal is one neuron?

No. A direction is a combination of many numbers in the residual stream, not a single neuron. The paper itself says the meaning of the direction remains unclear.

Does it work on every model?

The paper tested 13 open-source chat models from 1.8B to 72B parameters. The authors say the result may not generalize to untested models, especially larger or proprietary ones.

Is the refusal direction a safety problem or a safety tool?

Both readings exist. The authors present it as evidence that current safety fine-tuning is brittle, and also as an example of how understanding model internals can help control behavior.

Where can I learn more about interpretability?

Learn AI Alignment Theory has an Advanced course called Interpretability. Each lesson separates what is known from what is still open and lists its sources.

Get started

If you want to go from first principles to open research questions like this one, the About page shows how the courses are laid out. Each lesson gives every side of a debate with no verdict.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.