Circuit Tracing in Language Models Explained

Circuit tracing in language models explained in one line: it is a way to follow the steps a model took from one prompt to one answer, by drawing a graph of the internal parts that passed information along. It does not summarise what a model knows. It explains one output at a time.

If you want this and the rest of interpretability in short lessons, Learn AI Alignment Theory has an Advanced course called Interpretability, and you sign in with Google or an emailed code to start.

Circuit tracing in language models explained: what it means

Anthropic's methods paper, published on 27 March 2025, puts it this way: the team produces "graph descriptions of the model's computation on prompts of interest by tracing individual computational steps in a 'replacement model'." Two words need unpacking: the graph and the replacement model.

The field this belongs to is mechanistic interpretability: describing what happens inside a network in terms people can read. A feature is a readable building block the model uses. A circuit is the set of features and links that carry out one job. Circuit tracing finds the circuit behind a single answer.

How the replacement model makes tracing possible

The model's own neurons are hard to label, because one neuron often mixes several ideas (see superposition in neural networks). So the researchers train a stand-in. In the paper it replaces the model's multi-layer perceptron blocks with "cross-layer transcoders", components trained to imitate those blocks using sparse, more readable features. For the idea behind the part, see transcoders in interpretability, and for the related feature-finding tool, sparse autoencoders.

The paper reports that the stand-in can match the original model's outputs "in ~50% of cases". That is a useful number to keep in mind. In about half of the cases the replacement model alone would not give the same output, so what you read is a description of a proxy, not of the model itself.

What an attribution graph contains

The result is an attribution graph for one prompt. According to the methods paper, its nodes are active features, token embeddings from the prompt, reconstruction errors and output logits. The edges are linear effects between nodes. Because even a short prompt activates far too many features to read, the authors prune the graph to the nodes and edges that most contribute to the output at one token position.

A worked example: the Dallas question

The companion paper, "On the Biology of a Large Language Model", applies the method to Claude 3.5 Haiku. Its first case is the prompt "Fact: the capital of the state containing Dallas is", which the model completes with "Austin". The question the authors ask is whether the model really takes two steps (Dallas is in Texas, the capital of Texas is Austin) or uses a shortcut such as a memorised sentence. You can follow their reasoning in five steps:

1. Take one prompt and one output. Here, the prompt above and the word "Austin".
2. Trace back from the output. Find the features that pushed toward "Austin", then the features that fed those.
3. Read the graph. The authors report an internal step where the model represents "Texas", and say they can "see and manipulate" it.
4. Intervene. They inhibit the Texas features and switch on California features taken from a similar prompt about Oakland. The model then outputs "Sacramento". Swapping in Georgia features from a Savannah prompt gives "Atlanta".
5. Check what the graph missed. The paper says both two-hop reasoning and a shortcut are present in this example at the same time.

Step 4 is the point of the whole method. A graph is a hypothesis. If changing the middle feature changes the answer the way the graph predicts, that is evidence the model used that step.

Other findings the papers report

The biology paper reports more cases than Dallas. In one, the authors asked for a rhyming couplet and expected the model to improvise toward a rhyme. Instead they found that important components "were localized on the new-line token before the second line began", and they describe the model identifying possible rhyming words before it writes the line. This is a result for one prompt in one model, and the paper does not claim it holds for every task.

For a method that checks a model's internals instead of trusting its words, compare chain-of-thought faithfulness, which asks whether written reasoning matches the real reason for an answer. Circuit tracing is one way to look at the real computation; the two questions are related but not the same.

What circuit tracing cannot see yet

The authors are open about the limits, and they matter more than the headline cases.

• Most prompts do not work. The biology paper says the graphs give "satisfying insight for about a quarter of the prompts we've tried". The examples shown are success cases, and the paper says even those capture "only a small fraction of the mechanisms of the model".
• Attention is missing. The method does not explain how attention patterns are computed. You can see that a model looked back at a token, but not why.
• Dark matter. What the replacement model fails to reconstruct shows up as error nodes with no interpretable function. The paper says this is worse for complicated or unusual prompts, and that even the highlighted graphs have significant contributions from error nodes.
• A faithful stand-in is not assured. The methods paper says "we cannot guarantee that the replacement model has learned the same mechanisms" as the original, so it checks afterwards with perturbations. It calls its own validation "very coarse".

A related caution is in interpretability illusions: a neat explanation can still be wrong. Interventions, like the Texas to California swap, are the guard against that.

Why it matters for AI alignment

Testing a model from the outside shows what it does. Alignment worries often depend on why it does it: a model might give good answers for a bad reason, or learn a goal that only looks right on the training prompts. A method that reads the computation could give a second source of evidence that does not depend on what the model says about itself.

How far this will go is open. The papers describe the work as early, and they list the limits above in their own words. Neither paper claims that circuit tracing makes a model safe, and the biology paper says its examples are success cases. Treat it as a promising way to gather evidence about a model's internals, not a verdict on any model.

Frequently asked questions

What is circuit tracing in simple terms?

It is a method for mapping which internal features of a language model fed into each other to produce one specific output. The result is a graph for a single prompt.

What is an attribution graph?

It is the graph circuit tracing produces. Its nodes are active features, prompt tokens, reconstruction errors and output logits, and its edges are linear effects between them.

Why does it use a replacement model?

The original model's neurons are hard to read, so the researchers train cross-layer transcoders to stand in for parts of it. The cost is that the stand-in matches the original's outputs in about half of cases, per the methods paper.

Does circuit tracing show how a model really thinks?

Only partly. The authors say it gives satisfying insight for about a quarter of the prompts they tried and explains a small fraction of the model's mechanisms even in the best cases.

Where can I learn more about interpretability?

Learn AI Alignment Theory has an Advanced course called Interpretability. Each lesson separates what is known from what is still open and lists its sources.

Get started

If you want to go from first principles to open research questions like this one, the About page shows how the courses are laid out. Each lesson gives every side of a debate with no verdict.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.