Interpretability Illusions Explained: Why Explanations Mislead

Interpretability illusions explained in one line: a tool says it shows how a model works, you get a clean and believable story, and the story turns out to be wrong or true only on a narrow slice of inputs. If you read interpretability papers or run these tools yourself, you need to know how the illusions arise and how to test for them.

This post walks through four published cases, a checklist for testing an explanation, and why the problem matters for alignment. It assumes you know roughly what a neural network is. For a gentler start, read the posts on mechanistic interpretability for beginners and feature visualization first.

Interpretability illusions explained: what goes wrong

Interpretability tries to turn a huge pile of numbers into something a person can understand. A tool colors pixels, labels a neuron, or says "this direction stores the concept of a verb." You look at the output and feel that you understand.

An illusion is that feeling, not earned. The explanation looks right, but it does not describe what the model computes. Sometimes it describes the tool. Sometimes it describes the dataset. Sometimes it describes one corner of the model's behavior and fails everywhere else.

People are good at finding patterns and bad at noticing when a pattern was handed to them. Three habits make it worse.

• We pick examples that fit. If the ten images that excite a neuron all show dogs, you call it a dog neuron. You did not look at everything else it responds to.
• We accept correlation as mechanism. A signal that tracks a concept does not show the model uses that signal.
• We reward neat stories. A tidy picture is easier to publish, present and believe than a messy one.

The fix is not to distrust all interpretability. It is to ask what an explanation predicts, and then test that prediction.

Illusion 1: saliency maps that ignore the model

A saliency map colors the parts of an input that supposedly drove a prediction. For an image classifier you get a heat map over the picture, and it often looks like a sensible outline of the object.

In "Sanity Checks for Saliency Maps" (Adebayo and others, 2018), the authors found that relying only on visual assessment "can be misleading." They show that some existing saliency methods are independent of both the model and the data generating process. They explain the result with an analogy to edge detection, a technique that needs neither training data nor a model. Read the abstract on arXiv.

A worked example. A map for a bird photo outlines the bird's head. You randomize the model's weights and run the method again. If the outline is still there, the method is partly drawing edges, and the story "the model looks at the head" has no support yet.

Illusion 2: probes that read the probe, not the model

A probe is a small classifier trained on a model's internal activations to predict some property, such as the part of speech of a word. A high score tempts people to say the model "represents" that property.

Hewitt and Liang asked the obvious question in "Designing and Interpreting Probes with Control Tasks" (2019): does a good score mean the representation encodes the structure, or that the probe learned the task? Their answer is a control task that gives each word type a random output. A good probe should be selective: high accuracy on the real task and low accuracy on the control task that only the probe's own memory can solve. They found that popular probes on ELMo representations were not selective. Read the abstract on arXiv. To see probes used well, the post on linear probes covers the basics.

Illusion 3: neurons that seem to mean one thing

Looking at what excites a single neuron is a common way to name it. In "An Interpretability Illusion for BERT" (Bolukbasi and others, 2021), the authors describe an illusion where individual neurons may appear to encode a single, simple concept when they are encoding something far more complex. The same holds for linear combinations of activations.

They trace the source to geometric properties of BERT's embedding space and to the fact that common text corpora represent only narrow slices of possible English sentences. Their stated advice for researchers is to test hypotheses on multiple data sets. Read the abstract on arXiv.

The takeaway is plain. A neuron label made on one dataset describes that dataset. For why single neurons are often a poor unit of meaning, see the posts on superposition and sparse autoencoders.

Illusion 4: patches that work by a side door

Activation patching swaps internal activations from one run into another and watches the output change. It is a stronger test than a probe because it intervenes. It has its own trap.

Makelov, Lange and Nanda ("Is This the Subspace You Are Looking for?", 2023) show that two aims diverge: making the output behave as if a feature changed, and attributing that feature to a subspace. Counterintuitively, a subspace intervention can change the output by activating a dormant parallel pathway that is causally disconnected from the model's outputs. They show this in a distilled mathematical example, in the indirect object identification task and in factual recall, and present evidence that it is common in practice. Read the abstract on arXiv. The basics of the method are in the post on activation patching.

The lesson: an intervention that works shows the intervention has a causal effect. It does not show that the model's normal computation goes through that route.

How to test an explanation before you trust it

Treat every explanation as a hypothesis with predictions. The four cases above suggest this checklist for a paper or for your own experiment.

Each tool, the illusion it can produce, and the check that exposes it.

1. State the claim narrowly. "Neuron 31 fires on dog faces in this dataset" can be tested. "Neuron 31 is the dog detector" cannot.
2. Run a null control. Randomize the weights, use random labels or a random direction. If the result survives, it was not about the trained model.
3. Intervene, not only observe. Ablate, patch or steer, and check that the output changes as the story predicts.
4. Change the dataset. Try inputs unlike the ones that suggested the story. Illusions often break here.
5. Predict something new. A good explanation says what the model will do on a case you have not tried. Then try it.
6. List rival explanations. Ask what else would produce the same picture, and design a test that separates them.

A prompt to give yourself for a probe result: "If this probe is reading the model and not itself, what should happen on a control task, and when I edit the probed direction?" Write the answer down before you run the check.

Why illusions matter for AI alignment

Interpretability is often pitched as a way to look inside a model and check that it is safe. That hope depends on the tools being reliable. If a method can hand you a confident and wrong picture, then trusting it to find hidden goals is risky, and a false sense of understanding can be worse than knowing you do not understand.

The same question shows up across oversight: how do you check a system whose inner workings you cannot fully read? The posts on honesty in AI models and AI-written critiques for human evaluators look at neighbouring parts of it. Researchers hold different views on how far interpretability can go. None of the papers above says it is hopeless; they ask that results meet the standard of any empirical claim, with controls, interventions and predictions that could fail.

Frequently asked questions

What is an interpretability illusion?

It is an explanation of a model that looks convincing but does not match what the model computes. The tool, the dataset or a narrow set of examples creates the appearance of understanding.

Are all interpretability methods unreliable?

No. Each method has known failure modes, so you need controls and interventions before you trust a result. Tested findings can still be useful.

How can I tell if a probe result is meaningful?

Compare it with a control task, as Hewitt and Liang propose, and check that editing the probed direction changes the model's behavior. If neither supports the claim, treat the result as weak.

Does a successful activation patch show I found the mechanism?

Not by itself. It shows the intervention has a causal effect, but the effect can come through a route the model does not normally use.

Where can I study interpretability in a structured way?

Learn AI Alignment Theory has an Interpretability course among its Advanced courses, with short lessons, questions and hands-on activities. Its topics map follows the aisafety.com self-study topics.

Get started

To practice testing claims like these, Learn AI Alignment Theory teaches alignment from first principles to open research in lessons of about 8 minutes. Its Advanced level includes the Interpretability course, and each lesson lists its sources and separates what is known from what is still open. You sign in with Google or an emailed code. See also the posts on circuits in neural networks.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.