Path Patching Explained: Tracing Information Between Model Parts

Path patching explained in one line: it tests whether information travels along one chosen route between two parts of a neural network, instead of only asking whether a part matters at all. You change what a "sender" part says, let that change reach only a chosen "receiver" part, and watch whether the model's output moves.

If you want interpretability set in the wider alignment picture, Learn AI Alignment Theory teaches it in short hands-on lessons, including an Interpretability course at the Advanced level, with each lesson listing its sources.

Path patching explained: what it does in a transformer

A transformer is a stack of parts that read from and write to a shared stream of numbers. Attention heads and MLP layers each read the stream and add their result back, so later parts read what earlier parts wrote.

Path patching asks about one link in that chain. Does head A write something that head C reads, and does that link matter for the answer? It is a causal experiment: you intervene on one route and measure what changes.

Wang, Variengien, Conmy, Shlegeris and Steinhardt introduced the technique in their 2022 study of how GPT-2 small completes sentences like "When Mary and John went to the store, John gave a drink to" (arXiv 2211.00593). A later paper from Redwood Research by Goldowsky-Dill and colleagues (arXiv 2304.05969) generalized it to hypotheses about any set of paths in a network.

Why activation patching alone cannot show a route

Plain activation patching swaps one part's output from one run into another. If you patch head A and the answer changes, A matters. Our post on activation patching covers it, and its blunt side: a patch on A reaches everything downstream at once.

Heimersheim and Nanda's guide to the method puts it plainly: patching a component normally affects all components in later layers, while in path patching each patch affects only a single target component (arXiv 2404.15255). They note that this separates two structures that look identical in default patching. In one, A affects the output through B. In the other, A affects the output directly while B reads from A and boosts or cancels the signal. A direct connection between A and the output exists only in the second case.

The five steps of path patching

You need two inputs: an original one where the model does the behavior you study, and a new one that is similar but removes the information the behavior needs. You also pick a sender head and a set of receivers. Appendix B of the 2022 paper summarizes the recipe in five steps:

1. Gather activations on the original and the new input.
2. Freeze every attention head to its value on the original input, except the sender, which takes its value from the new input.
3. Run a forward pass on the original input with those frozen and patched parts. MLPs and layer norm are recomputed.
4. Save what the receivers see in that pass.
5. Run a last pass on the original input, patching only the receivers to the saved values.

Because every other head is frozen in step 3, any influence that passes through a second head is removed. What reaches the receivers comes only from the sender's direct paths, through the residual stream and MLPs. The paper scores the result as the change in logit difference compared with the unpatched model, averaged over more than 200 pairs of inputs.

A worked example from the paper: who feeds the output?

The 2022 paper's task is indirect object identification. In "When Mary and John went to the store, John gave a drink to", GPT-2 small should predict "Mary". For the new input the authors replace the names with three random names, which removes the information the task needs. Then they trace the circuit backward in two sweeps.

1. Sweep 1: which heads write directly to the output? For every head at the final position, they patch the path from that head to the logits. Only a few heads in the last layers had a large effect. Patching heads 9.6, 9.9 and 10.0 caused a large drop in logit difference. Patching 10.7 and 11.10 caused a large increase, so those two push against the right answer.
2. Look at what the first group does. The authors found these heads attend strongly to the indirect object, with an average attention probability of 0.59, and named them Name Mover Heads.
3. Sweep 2: who feeds the Name Movers? They repeat the patch for every head, this time on the path into the Name Movers' query vectors. That sweep points to a further group, among them the S-Inhibition Heads, whose job in the paper's account is to stop the Name Movers attending to the repeated name.

Step by step this builds a circuit: 26 attention heads grouped into 7 classes. For the general idea of circuits, see our post on neural network circuits.

Testing a whole explanation with path patching

The Redwood paper changes how the method is used. Instead of tracing one path at a time, you state a hypothesis: these paths are important and all other paths are not. Path patching then feeds the unimportant paths a counterfactual input and measures how much of the model's behavior survives. A score of 0% means the hypothesis is as bad as claiming nothing matters, and 100% means the loss matches the original model on every example.

The authors test this on induction behavior in a two-layer attention-only transformer, with one previous token head and two induction heads (compare our post on induction heads). The minimal textbook explanation did not explain much:

• The direct path to the induction heads' values explained 64.1%, the direct path to the queries 48.7%, and the previous token head's path to the keys 53.3%.
• All three together explained only 28.2%.
• After adding positional paths, the combined hypothesis explained 48.6%, and a longer window of three tokens raised it to 55.2%.

The lesson is not the numbers. A story about a circuit can sound complete and still leave about half of the behavior unexplained, and path patching says by how much. When a hypothesis fails, path patching attribution shows where, so you can refine the claim and test again.

Path patching vs attribution patching

Path patching is exact but costly: each sender and receiver pair needs its own forward passes. Attribution patching trades accuracy for speed by using gradients to estimate what patching would do, so one backward pass scores many parts. Our post on attribution patching covers where its estimate can mislead.

Heimersheim and Nanda suggest using both: start with low-granularity patching, increase granularity, and finish with path patching to test which components interact. The fast method tells you where to look; path patching checks whether you were right.

What path patching cannot tell you

The new input defines the question

A positive result only shows you found parts that deal with the difference between the two inputs. Heimersheim and Nanda advise considering several sets of corrupted prompts. The Redwood paper goes further and treats the dataset as part of the claim: a good explanation on one dataset may not transfer to another.

Other parts may step in

The 2022 paper found Backup Name Mover Heads that copy names only when the regular Name Movers are knocked out. A weak effect from one head can therefore hide a part that would matter in another situation. See also our post on interpretability illusions.

Small models, narrow tasks

The IOI authors say they still do not understand several components, including the S-Inhibition Heads' attention patterns and the effect of MLPs and layer norms. They also note GPT-2 small is orders of magnitude smaller than state-of-the-art models, and that scaling the approach is a future challenge.

Why path patching matters for AI alignment

Behavior tests only show what a model did on the inputs you tried. Path patching adds causal evidence about internal routes, so a claim like "the model uses this mechanism" can be checked instead of assumed. Inner alignment questions, such as whether a model has learned the goal we meant, are about what happens inside, and tracing routes is one tool for looking.

Mind the limits above: evidence from GPT-2 small on a narrow task is not evidence about a frontier model, and nothing here shows a model is safe. Each lesson in the app separates what is known from what is still open, a good habit for any interpretability claim. For a wider tour, start with mechanistic interpretability for beginners.

Frequently asked questions

What is path patching in simple terms?

It is an experiment that changes the output of one model part and lets that change reach only one chosen later part. If the final answer shifts, the route between them carried information the model used.

How is path patching different from activation patching?

Activation patching changes a part for everything downstream. Path patching lets the change affect a single target component, so an effect can be credited to a specific connection.

Who introduced path patching?

Wang and colleagues introduced it in their 2022 study of indirect object identification in GPT-2 small. Goldowsky-Dill and colleagues later generalized it to hypotheses about any set of paths.

Is path patching better than attribution patching?

It is more exact but needs more compute. Attribution patching is a fast approximation, so researchers can use it to find candidates and then confirm them with real patches.

Does path patching prove a model is safe?

No. It shows which routes matter for a tested behavior on tested inputs, in the models studied so far. It says nothing about everything else a model might do.

Get started

The lessons run about 8 minutes each and include questions and hands-on activities. You sign in with Google or an emailed code, and each lesson lists its sources so you can read the original research next. The About page explains how it works.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.