Mechanistic Anomaly Detection: Same Output, Different Reason

What is mechanistic anomaly detection?

Mechanistic anomaly detection is the problem of telling whether a model reaches an output for its usual reason or for a different one, even when the output looks the same. Ordinary anomaly detection asks whether an input looks strange. This asks whether the model's reasoning looks strange. The name and framing come from the Alignment Research Center (ARC), in a post by Paul Christiano that describes joint work with Mark Xu. The post calls itself an informal description of ARC's current research approach, so treat what follows as a research direction, not a finished tool.

Learn AI Alignment Theory teaches how to make AI systems do what we intend, from first principles to open research, in short hands-on lessons that give every side of every debate.

The idea matters because some alignment failures produce outputs that look fine. If you only check the output, you see nothing. If you could check the reason, you might.

The diamond and cameras example

ARC starts from a story. A diamond sits in a vault, watched by cameras. An ML system predicts what the cameras will see over the next hour. You want to tell apart two cases: the model predicts the diamond will really stay in the vault, or the model predicts that someone will tamper with the cameras so the diamond only appears to stay.

This is the setup behind the eliciting latent knowledge problem. Mechanistic anomaly detection is ARC's proposed route to it. The plan is to find the normal reason the diamond appears on camera when it really is there, then check, on each new input, whether the same reason applies.

How mechanistic anomaly detection works, step by step

ARC's outline has a few moving parts. Here is the shape, in the order a researcher would follow it.

1. Collect trusted cases. ARC assumes a dataset of situations where the diamond appears to remain in the vault, and where that appearance is always because it really does remain. The model predicts well on this data.
2. Find the normal reason. Ask why this computation, run on this input distribution, produces such a specific pattern of predictions. ARC wants an explanation in terms of the model's weights and the properties of the inputs. It calls this a mechanistic explanation, as opposed to the teleological one, which is just that the model was trained to match reality.
3. Test a new input. When the model again predicts that the diamond appears to remain, ask whether the normal explanation still covers it, or whether something different, or a coincidence, is going on.
4. Flag the difference. If the usual reason does not apply, treat the output as suspect.

ARC states the general problem more formally. Take a function that combines the model with some property of its output, and a training set where that value varies. For a new input, ask how much of its deviation from the average is captured by the normal reasons that explain the variance on the training set, as opposed to new reasons. The post stresses that this differs from spotting an outlier input. A model can behave oddly on harmless training examples, or normally on strange inputs.

A worked example you can follow

Take ARC's heist. In the trusted data, the diamond looks stable from frame to frame. The normal explanation is that the cameras pick up photons reflected off a stable arrangement of atoms, and the arrangement is stable, so the images match.

Now run three new cases and ask the one question: does that explanation still apply?

• The diamond stays. The cameras pick up photons from the same stable atoms. The explanation applies. No flag.
• A robber swaps in a fake between two frames. The image barely changes, but the stability is now because the robber copied how the diamond looks. The atoms were not stable. The explanation does not apply. Flag.
• The robber tampers with the sensors. The cameras are no longer capturing photons from the vault. The explanation does not apply. Flag.

In all three, the output is "the diamond is still there". Only the reason differs. ARC adds a striking remark: if the robber acts so that the diamond keeps appearing for the normal reason, it seems plausible to call that action "not stealing the diamond".

Why it matters for deceptive alignment

ARC also points at a harder case. Picture a model that behaves well in training because it needs to in order to survive, then behaves differently once deployed. ARC's hope is that although both behaviors use the same parts of the model, the variance is driven by different mechanisms: honest beliefs in training, strategic answers later. A detector that separates them could flag a treacherous turn, and the system could then be shut down and retrained on the offending case.

ARC is careful about the limit. In some high-stakes situations a temporary shutdown could itself cause a catastrophe, or you might have no way to supervise the flagged cases even after the flag. For the background worry, see our posts on deceptive alignment and scheming and sleeper agents.

Testable versions: backdoors, language models and toys

The post lists research problems you can try without solving the whole thing.

• Backdoor attack detection. Train a model on a clean dataset and also to output large values on a backdoor dataset. Then detect inputs where the output is large because of the backdoor. ARC likes this because two clearly different mechanisms drive behavior and it is easy to tell whether a method works. It separates this from trojan detection, where you must decide whether a model has a backdoor at all. Attack detection should be far easier, and trojan detection is likely very difficult.
• Natural mechanism distinctions. Find language-model behaviors that can arise two ways, for example a story character answers a factual question wrongly because the model does not know the answer, or because the model thinks the character would lie. Then check whether a method can tell the two apart.
• Toy versions of ELK. ARC sketches a gridworld with a pushable diamond, a pressure plate and two cameras, and a 100 by 100 game of life grid with two sensors that fire on a glider crossing the grid.

ARC says it hopes interpretability tools can succeed at tasks like these, since they give a way to check progress. If you want that background, read mechanistic interpretability for beginners and linear probes explained.

What "explanation" means, and what is still open

The hardest word in the plan is "explanation". ARC says it has no adequate notion yet. Its current attempt is the heuristic arguments line of work. In Formalizing the presumption of independence, Christiano, Neyman and Xu introduce heuristic estimators, set out properties they should satisfy, and pose the main open problem: is there an estimator that formalizes valid uses of the presumption of independence without accepting spurious arguments? ARC also names Redwood Research's causal scrubbing as a different candidate that may fill the same role.

ARC lists three problems it works on: formalizing heuristic arguments, using them for mechanistic anomaly detection, and finding the explanations in real models. Of the last, it says it seems quite difficult and there is a good chance it will not be possible, by this plan or any other. Also note that ARC is focused on theoretical approaches for the worst case, while it says empirical work could help or yield a practical approach for realistic networks. The post is a plan from 2022, so read it as a statement of intent, not a record of results.

Frequently asked questions

What is mechanistic anomaly detection in simple terms?

It asks whether a model reaches an answer for its usual reason. The same output reached by a different internal route counts as an anomaly, even if the output looks normal.

How is it different from ordinary anomaly detection?

Ordinary anomaly detection looks for unusual inputs. ARC's version looks at how the model handles the input, so a normal-looking input can be flagged and a strange one can pass.

Is it related to eliciting latent knowledge?

Yes. ARC's post is a follow-up to its Eliciting Latent Knowledge report, and the diamond and cameras example comes from it. The aim is to tell a real diamond from a tampered camera.

Does it work today?

The source does not say it does. ARC describes research problems, a plan and open difficulties, including that finding good explanations may not be possible.

Get started

The Scalable Oversight course in Learn AI Alignment Theory has a Latent Knowledge lesson, and the Interpretability course covers the tools this plan leans on. Lessons run about 8 minutes, list their sources and give every side of every debate. You sign in with Google or an emailed code. Read what the courses cover for the overview.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.