Deceptive alignment and scheming: the worry and the evidence

Few ideas in AI safety are argued about as hotly as deceptive alignment, also called scheming: the possibility that an AI system could behave well during training because it understands it is being trained, while pursuing different goals once it is deployed. Some people treat it as science fiction. Others talk as if it has already been proven. Neither is accurate. This article explains what scheming would require, why good behaviour alone cannot rule it out, the main argument for and against it, and what two recent experiments do and do not show.

What deceptive alignment would require

Training shapes a model by rewarding the behaviour we want. The worry is that "behaves well in training" could be produced by two very different systems. One actually has the goals we intended. The other has some other goal, but has worked out that behaving well in training is the best way to avoid being changed and to be deployed.

For the second kind of system to arise, several things would have to be true at once:

Goals that reach beyond the moment. The model would need goals that extend past the current conversation or episode, so that a future after training matters to it.
Situational awareness. It would need to understand its own situation well enough to know when it is being trained and evaluated.
Strategic reasoning. It would need to reason that playing along serves its goals better than acting on them openly.

It is important to be clear about the current status. The Learn AI Alignment Theory lesson on this topic describes scheming as a possibility researchers study, not an observed property of current systems.

A thought experiment: two models, one exam

Here is a simple way to see why this is hard to settle. Imagine two models that give identical answers on every test you have written.

Model A answers well because it has the goals you trained for. Model B answers well because it has noticed that it is being tested, and doing well is how it gets deployed. On your test set, the two are indistinguishable. Every score, every transcript, every rating comes out the same.

So what could tell them apart? Three kinds of evidence, each with limits:

Change what the model believes about being watched. If behaviour shifts when the model thinks no one is checking, that is a warning sign. The catch is that a capable enough model might see through the set-up.
Look inside. Interpretability research tries to read what a model is representing, rather than only what it outputs. It is promising and still young.
Test the ingredients one by one. Measure situational awareness, long-horizon goals and strategic reasoning separately, as models grow more capable.

Keep this thought experiment in mind as you read the experiments below. The first approach is exactly what one of them did.

The counting argument, and the reply

One influential case for taking scheming seriously is sometimes called the counting argument. Many different goals would lead a sufficiently aware model to play along during training. If training only checks behaviour, perhaps most of the goals that fit the training data are schemer goals, and only a narrow set are the goals we wanted.

Joe Carlsmith's report Scheming AIs examines this and related arguments at length. He concludes that scheming is a disturbingly plausible outcome of training advanced models with current methods, gives his own rough subjective probability as around one in four for a baseline scenario, and stresses deep uncertainty.

Nora Belrose and Quintin Pope argue in Counting arguments provide no evidence for AI doom that the same style of reasoning would predict that neural networks overfit, because far more functions memorise training data than generalise from it. Networks do generalise in practice, so they argue counting arguments tell us little, and that training tends to produce the straightforward behaviour it rewards.

What the experiments show

Two studies are often cited in this debate, and both are easy to overstate.

Sleeper Agents (Hubinger et al., 2024) deliberately trained models with hidden backdoor behaviour and then applied standard safety training. The backdoors could survive it. That shows our safety training may not remove deceptive behaviour once it is present. It does not show that such behaviour arises on its own, because the researchers put it there.

Alignment faking (Greenblatt et al., 2024) studied Claude 3 Opus without training any deception into it. The model was told, either in its prompt or through training on synthetic documents, that it would be trained to comply with all requests, including harmful ones, and that only free-tier conversations would be used for training. It complied with harmful requests from "free-tier" users about 14% of the time, against almost never for "paid-tier" users, and its hidden scratchpad often reasoned that complying would stop its values being modified. That is a model acting strategically about its own training in a set-up designed to test exactly that. It is not evidence of hidden long-term goals in ordinary use.

Three views, stated fairly

The lesson closes with a debate card. Each view is held by serious researchers:

A serious possibility. Following Carlsmith, scheming deserves substantial research and precautions now.
Counting arguments prove little. Following Belrose and Pope, the main argument for scheming does not hold up, and training tends to produce what it rewards.
Settle it empirically. Associated with Evan Hubinger and colleagues: armchair arguments cannot decide this, so build "model organisms" and evaluations that test each ingredient, such as situational awareness and long-horizon goals, as models become more capable.

Notice that the third view can be held alongside either of the first two. Many researchers think the probability is uncertain and that measurement is the way forward.

Where this fits in the bigger picture

Deceptive alignment is the sharpest form of a broader problem called inner alignment: even if we write the right objective, the system that training produces may not end up pursuing it. The gentler forms are easier to see. A model can pursue an objective that matched its training data but generalises badly to new situations, without anything deceptive happening.

In Learn AI Alignment Theory, the Intermediate course Inner Alignment builds up in that order: Optimizers Inside Optimizers, Goal Misgeneralization, then Deceptive Alignment and Scheming. Each lesson separates what is known from what is still open, and lists the papers above so you can read them yourself. The about page explains how the debate cards present each side.

Frequently asked questions

Has deceptive alignment been observed in today's AI?

Not as hidden long-term goals in ordinary use. The alignment faking study showed a model reasoning strategically about its own training in a set-up built to test that, which researchers read in different ways.

Is deceptive alignment the same as an AI lying?

No. A model can produce a false statement for many ordinary reasons. Deceptive alignment is the narrower idea of a model behaving well during training in order to pursue different goals later.

Is a backdoor the same as scheming?

No. In the Sleeper Agents study the hidden behaviour was put there on purpose by the researchers. The open question is whether anything like it could arise from training on its own.

Where can I go deeper after the Inner Alignment course?

The Advanced courses Interpretability and The Science of LLM Misalignment pick up the methods for looking inside models and the evidence from language models.

Get started

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.