Transcoders in Interpretability Explained Simply

If you searched for transcoders in interpretability explained simply, here is the short version. A transcoder is a small, wide model trained to stand in for one part of a large neural network, so that researchers can read what that part does. It turns a tangled step of computation into a list of pieces that people can label and test.

This post covers what a transcoder copies, how it differs from a sparse autoencoder, how it feeds circuit maps, a worked example from Anthropic's own paper, and the limits its authors admit. No math is needed. If the field is new to you, start with mechanistic interpretability for beginners.

Transcoders in interpretability explained simply: the core idea

Interpretability is the work of finding out what happens inside a trained model. A transcoder is one of its tools. In the words of Dunefsky, Chlenski and Nanda, transcoders "seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer" (arXiv 2406.11944).

Read that slowly. The original layer is dense: lots of its neurons are active at once. The copy is wide and sparse: it has many more units, and only a few are active for any one input. You then study the copy, which is easier to read than the original.

Why MLP layers are hard to read

A language model is built from repeated blocks. Each block has attention, which moves information between positions in the text, and an MLP, which transforms the information at each position. The MLP is the part that is hard to describe.

The same paper explains why. MLP neurons are often polysemantic, meaning one neuron activates on many unrelated concepts. Sparse autoencoders get around that by looking at features instead of neurons, but the authors note that SAE features "tend to be dense linear combinations of many neurons". To see how one feature leads to a later one through the MLP, you would then have to account for an infeasible number of neurons and their nonlinearities. For the first problem, see our post on superposition.

Transcoders vs sparse autoencoders

The two tools look alike and are trained alike. The difference is the target. Paulo, Shabalin and Belrose put it in one line: transcoders "are trained to reconstruct the output of a component of a deep network given its input" (arXiv 2501.18823). A sparse autoencoder rebuilds the same activations it reads.

So an SAE gives you a snapshot of what is present at one point. A transcoder gives you a stand-in for a step, the computation that turns one thing into the next. Our post on sparse autoencoders covers the snapshot side.

Which one reads better? Paulo and colleagues trained both on the same model and data and found that "transcoder features are significantly more interpretable". Dunefsky and colleagues, testing on GPT2-small, Pythia-410M and Pythia-1.4B, found transcoders equal to or better than SAEs on the sparsity-accuracy tradeoff, with the gap seeming to widen on larger models. They add a caveat: with limited hyperparameter sweeps, a different setting could have let SAEs catch up. Treat this as promising, not settled.

From transcoders to circuits

The payoff is circuits. Because the transcoder replaces the MLP with sparse features, the path through the MLP becomes links between named features. Dunefsky and colleagues report that the resulting circuits "neatly factorize into input-dependent and input-invariant terms": one part depends on the prompt, and one part is a property of the model's weights that holds for every prompt. That second part is what lets researchers say something about a feature without any example text.

They tested this with blind case studies. They picked nine features at random in a layer 9 transcoder of GPT2-small and tried to work out what each feature does while the prompt tokens were hidden, so that nobody could read the answer off the examples. A researcher who has already seen the examples can bias the analysis toward the story they expect. For why neat explanations can mislead, see our post on interpretability illusions. For the wider idea of a circuit, see circuits in neural networks.

Attribution graphs, a worked example

Anthropic's circuit tracing work takes the idea further. It builds a replacement model in which "cross-layer transcoders" stand in for the MLPs, then traces each computational step to draw an attribution graph for one prompt. Work through the paper's factual recall case study:

1. Pick a prompt. Fact: Michael Jordan plays the sport of. The model completes it with "basketball", with 65% confidence in the paper.
2. Draw the graph. Each node is a feature. Each edge says how strongly one feature pushed another.
3. Group similar features. The researchers merge related features into supernodes so the graph can be read.
4. Read the paths. The paper reports two main ones. One starts at the "plays" and "sport" tokens and promotes "say a sport" features. The other starts at "Michael Jordan and other celebrities" and promotes basketball features, with a positive edge to the basketball answer and a negative edge to the football answer.
5. Test it. A graph is a hypothesis. The authors check graphs by inhibiting or exciting features and seeing whether the output and other features move as predicted.

Two paths, one about the kind of answer wanted and one about who the person is, came out of a single prompt. That is the kind of statement a graph lets you make and a pile of neurons does not. It is one prompt in one small model, and the paper's own validation is a different matter, covered next.

What transcoders cannot tell you

Both papers are open about limits.

• The copy is an approximation. Dunefsky and colleagues write that the error "may lose key information", and find transcoders about as unfaithful as SAEs by cross-entropy loss. Anthropic's graphs carry error nodes for the part of each MLP output the transcoder leaves unexplained.
• Attention is treated as fixed. Both methods freeze or hold attention patterns, so they say how information is used once attention has moved it, not why the model looked at a given position. Anthropic lists missing attention circuits as a limitation.
• A good fit is not proof of the same mechanism. Anthropic calls this mechanistic faithfulness: even a transcoder that rebuilds the activations well might get there by different means, and the authors say they cannot guarantee it. Their perturbation tests, they note, are coarse.
• Few case studies. Dunefsky and colleagues say their results "would be stronger with more systematic analysis".
• Up-front cost. Anthropic notes that training a cross-layer transcoder can incur significant up-front cost and effort.

Researchers also disagree about how far sparse features can take interpretability. Some treat them as a route to reading models closely. Others doubt that a sparse dictionary captures how a model computes. This post takes no side.

Why transcoders matter for AI alignment

Training shows you behavior. Interpretability tries to show the steps behind it. That matters for the questions in our post on outer vs inner alignment: if a model's trained goal and its real goal can differ, a way to inspect its steps is useful. The same holds for deceptive alignment and scheming, where outputs alone may not tell you enough.

A transcoder graph is still evidence, not a safety guarantee. By the authors' own account it is a hypothesis about one prompt that needs testing, and the tools are young.

Frequently asked questions

What is a transcoder in simple terms?

It is a wide, sparse model trained to copy what one component of a network does, such as an MLP layer, predicting the component's output from its input. You study the copy because it is easier to read.

How is a transcoder different from a sparse autoencoder?

A sparse autoencoder rebuilds the activations it reads. A transcoder predicts a component's output from that component's input, so it stands in for a step of computation.

What is a skip transcoder?

Paulo and colleagues propose adding an affine skip connection to the transcoder. They report lower reconstruction loss with no effect on interpretability.

Do transcoders fully explain how a model works?

No. The copy has error, attention is held fixed, and a good fit does not prove the same mechanism. The authors present graphs as hypotheses to test.

Get started

Learn AI Alignment Theory has an Advanced course called Interpretability, alongside Inner Alignment and Scalable Oversight, where these ideas sit in context. Lessons run about 8 minutes, list their sources, and separate what is known from what is still open. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.