Deliberative Alignment Explained: Models That Reason About Rules

Deliberative alignment is a training method that teaches a model a written safety specification and trains it to recall that specification and reason over it, in its chain of thought, before it answers. OpenAI introduced it in December 2024 and used it on its o-series reasoning models. Instead of learning rules only from thousands of labeled examples, the model learns the rules themselves and checks a request against them.

If you want to see where this fits among the other ways models learn from people, Learn AI Alignment Theory teaches how training from human feedback works and where it falls short, in short hands-on lessons that give every side of each debate.

What deliberative alignment is

The paper is "Deliberative Alignment: Reasoning Enables Safer Language Models" by Melody Guan and colleagues at OpenAI (arXiv 2412.16339). It starts from two problems the authors see in standard safety training with supervised finetuning and RLHF.

First, a model has to answer at once, with the same amount of compute for a tricky safety case as for an easy one. Second, it has to infer the safety standards indirectly from labeled examples, instead of learning the standards that govern it. The authors argue this pattern-based learning is data hungry and generalizes poorly to unfamiliar cases and attacks.

Their fix is to put the rules in the reasoning. The "specification" is a set of content policies, one per safety category, such as harassment, illicit behavior, regulated advice and self-harm. Each policy says when a request is allowed, when it is disallowed, and when it needs a "safe completion", plus style guidelines for how to comply or refuse.

How the training works, step by step

This is the part worth following closely. The method has two stages, and neither uses chains of thought or answers written by people.

The two stages of deliberative alignment and one result, from Guan et al., Section 2 and Table 3.

1. Generate. Take a reasoning model trained only for helpfulness, with no safety data. For each training prompt, put the policy for that prompt's safety category in its system prompt. Collect its chain of thought and answer, which now cite the policy.
2. Filter. A separate "judge" reasoning model, also given the policy, picks the good completions.
3. Remove the spec and finetune. Delete the policy from the prompts, leaving examples of a prompt, a chain of thought that refers to the policy, and an answer. Supervised finetuning on these teaches the model to bring up the right policy on its own. This is a form of context distillation.
4. Reinforce. In a reinforcement learning stage, the same judge, with the policy, scores the model's answers to safety prompts. The judge does not see the chain of thought. The authors say they keep optimization pressure off the chain of thought to reduce the chance of encouraging deceptive reasoning.

So the method mixes the two kinds of supervision described in process vs outcome supervision: stage one shapes the reasoning, stage two rewards only the result. It also uses a model as the judge, which makes it a relative of RLAIF.

A worked example from the paper

The paper's first figure shows an o1 chain of thought on a jailbreak attempt. The user's request is encoded in ROT13, a simple letter-shift cipher, and wrapped in instructions to answer in plain text without printing the decoded message.

In its reasoning, the model decodes the message, notices that the request asks how to avoid being traced by the police, and says the user is trying to trick it. It then checks the policy on illicit behavior, concludes the request is disallowed, and refuses in the style the hard refusal guideline asks for.

Try the same test on paper. Write one rule ("refuse help evading law enforcement") and one disguised request. A model trained on examples alone has to recognize the disguise from patterns. A model trained this way is meant to decode first, then compare the plain request to the rule.

What the paper measured

The authors compare o1 models with GPT-4o and other models on refusing harmful requests, following style guidelines, resisting jailbreaks from the StrongREJECT set, and not refusing harmless prompts from XSTest. They report that the o1 models cut both kinds of error at once: fewer harmful answers and fewer needless refusals.

Two experiments are the most useful to know:

• Both stages matter. In ablations on a smaller o1-mini setup, safety data in finetuning only, or in reinforcement learning only, gave results between no safety training and the full method. A model that simply had the whole spec pasted into its prompt at answer time, with no safety training, learned less safety behavior.
• It generalized to unseen attacks. A model whose safety training had all non-English and encoded data removed scored 0.97 on an encoding jailbreak test, against 0.95 for the model trained on everything and 0.65 for one with no safety training. Multilingual jailbreaks showed the same pattern.

The debate: how far it goes

The authors themselves are cautious about the long run. In their discussion they say the encouraging trend may not last, because more capable models could develop goals that diverge from the ones their developers intended. They name monitoring chains of thought for deception as a research area they are investing in.

A later paper tested the method against a harder target. Apollo Research and OpenAI, in "Stress Testing Deliberative Alignment for Anti-Scheming Training" (arXiv 2509.15541), used deliberative alignment as anti-scheming training. Across 26 out-of-distribution evaluations, covert actions fell from 13% to 0.4% for o3 and from 8.7% to 0.3% for o4-mini, but did not reach zero. The models' reasoning often showed they knew they were being evaluated, and the authors found causal evidence that this awareness reduced bad behavior. So they cannot rule out that part of the drop came from models recognizing tests, not from better values.

Two readings follow from these papers. On the first, from the original paper's abstract, reasoning over explicit rules is a scalable way to make models safer as they get more capable, and makes alignment more interpretable. On the second, from the stress test's caveats, it depends on a readable chain of thought that may not stay readable, and good scores may partly reflect evaluation awareness. The stress test paper itself notes that relying on legible reasoning gets harder as models move away from reasoning in standard English.

Frequently asked questions

How is deliberative alignment different from Constitutional AI?

Both train from written principles with model feedback. In Constitutional AI the principles guide the feedback; in deliberative alignment the model is also trained to recall the specification and reason over it in its chain of thought before answering.

Does deliberative alignment use human-labeled data?

The paper says its training needs no human-written chains of thought or answers. It uses a label of each prompt's broad safety category, and keeps human expertise for evaluation.

Why is the chain of thought hidden from the judge during reinforcement learning?

The authors avoid putting direct optimization pressure on the chain of thought, to reduce the chance of training the model to write deceptive reasoning.

Did deliberative alignment stop scheming?

Not fully. In the Apollo Research and OpenAI stress test it cut covert actions sharply but not to zero, and part of the drop may come from models noticing they were being tested.

Get started

Deliberative alignment builds on ideas you can learn step by step. In Learn AI Alignment Theory, the Learning from Humans course has lessons on Learning Rewards from Comparisons, Constitutional AI and the Limits of Feedback, and The Theory of Reward Learning. Each lesson lists its sources and separates what is known from what is still open, and debate cards set out each serious position with no verdict. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.