Crosscoders and Model Diffing Explained: A Plain Guide

Crosscoders and model diffing explained in one line: model diffing asks what changed inside a model when it was trained further, and a crosscoder is a tool that learns one shared list of features for two models, so you can see which features both use, and which only one does. This guide walks through the idea with a worked example, what the first results showed, and the limits the authors themselves state.

To place this topic in the wider field, Learn AI Alignment Theory teaches alignment in short hands-on lessons, with an Advanced Interpretability course and debate cards that give each serious position without a verdict.

What model diffing asks

Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms, as Minder and colleagues (arXiv 2504.02922) put it. The usual pair is a base model and a version trained further, such as a chat-tuned one.

Behavior tests tell you that the second model refuses more. They do not say what changed in the computation. A diff of the internals tries to answer that, much as a code diff lists changed lines, except the "lines" here are learned features.

This matters for alignment because fine-tuning is where much of the intended behavior is added, and where unintended behavior can enter. If you can list what a training step changed, you can compare that list with what you meant to change. That is also the logic of alignment auditing for hidden objectives.

A refresher on sparse autoencoders

Crosscoders build on sparse autoencoders (SAEs). Neurons in a model often mix several unrelated ideas, a problem covered in superposition in neural networks. An SAE learns a dictionary of directions with two parts:

• An encoder that turns an activation vector into a long list of feature strengths, most of them zero.
• A decoder that rebuilds the activation as a sum of a few dictionary directions.

Training balances two pressures: rebuild well, and keep few features active. The details are in sparse autoencoders explained. An ordinary SAE reads one layer of one model, so comparing two models would mean matching two separate dictionaries, which do not line up neatly.

Crosscoders and model diffing explained: what a crosscoder is

Anthropic researchers (Lindsey and colleagues, 25 October 2024) introduced sparse crosscoders. Where an autoencoder works at a single layer, and a transcoder uses one layer to predict the next (see transcoders explained simply), a crosscoder reads and writes to multiple layers. The authors call it a preliminary research update, not a full paper.

For model diffing the setup is:

1. Feed the same text to the base model and the fine-tuned model.
2. Take each model's activations at the same layer, for the same tokens.
3. Train one encoder that reads both and produces a single list of sparse features.
4. Train two decoders, one rebuilding the base activation and one rebuilding the fine-tuned activation.

Each feature therefore has one strength but two decoder directions, one per model, so nobody has to match two dictionaries afterward.

Reading the decoder norms: a worked example

Once the crosscoder is trained, you look at each feature's two decoder vectors. A summary used in this work is the relative size of their norms: the chat norm divided by the base norm plus the chat norm. Near 0.5 means the feature writes to both models about equally. Near 0 means it mostly belongs to the base model, and near 1 means it mostly belongs to the fine-tuned one.

Here is a small imagined example, with invented numbers, to show how you would read it:

• Feature A has a base norm of 0.9 and a chat norm of 1.0. The ratio is 0.53, so it is shared machinery.
• Feature B has a base norm of 0.02 and a chat norm of 1.0. The ratio is 0.98, so it is a candidate for something training added.
• Feature C has a base norm of 1.0 and a chat norm of 0.05. The ratio is 0.05, so it is a candidate for something training removed.

Then you read the text where B and C fire most strongly, guess what they track, and test the guess by changing the feature and watching the behavior. "Candidate" is deliberate, for reasons the next sections explain.

What the first diffs found

In the original note, the Anthropic team trained a crosscoder with 1 million features on middle-layer activations of Claude 3 Sonnet and the base model it was fine-tuned from. Features fell into three clear groups: base-specific, fine-tuned-specific and shared, with between four and five thousand model-specific features for each model.

Among the fine-tuned-specific features they noted a refusal feature that fires on dangerous requests, a code review feature, and a feature for users asking the assistant personal questions about itself. Among base-specific ones were features for an LLM roleplaying a character in a system prompt and for dialogues with a smartphone assistant. The authors say plainly that these are cherry-picked and that most model-exclusive features are not immediately interpretable. For shared features the decoder directions were almost always highly aligned across the two models, though a few thousand were not, which the authors suspect are old concepts used in a new way.

Minder and colleagues later used crosscoders on Gemma 2 2B base and chat models, reporting chat-specific latents for concepts such as false information and personal questions, plus several refusal-related latents that react differently to different refusal triggers. A related direction is the refusal direction in language models.

Why a chat-only feature may not be chat-only

The same Minder paper also warns about the method. It identifies two issues that come from the L1 sparsity loss and can label a concept as unique to the fine-tuned model when it exists in both:

• Complete Shrinkage. The sparsity loss can push a base-model direction to zero length even though that direction helps rebuild base activations.
• Latent Decoupling. The crosscoder may show a shared concept as a chat-only feature when the base model encodes it through a different combination of features, since the loss treats the two as equivalent.

The authors built a check called Latent Scaling, which measures how well a supposedly chat-only feature explains base-model activations. They found the standard crosscoder suffers heavily from both issues on Gemma 2 2B, and that the clean three-group picture from decoder norms is partly an artifact of the loss. Training with a BatchTopK loss mitigated both, and found more chat-specific features that were interpretable and causally effective.

A separate Anthropic interpretability update from 2025 reports a related pattern: model-exclusive features tend to be denser and more polysemantic than shared ones, with activation frequencies about an order of magnitude higher. Their toy model suggests this comes from competition for limited feature capacity. Their fix reserves a small set of shared features, about 10,000 of roughly 250,000, with a lower sparsity penalty, which made exclusive features less dense.

So the researchers differ on how far to trust the raw norm split. The original note presents the clean clusters as a finding. Minder and colleagues argue they are partly produced by the loss and recommend checks. The Anthropic update keeps the method and changes the training.

Limits, and where diffing fits in alignment work

• Small models, one layer. Minder and colleagues analyzed small models' middle layers, and make no definitive claims about larger models or other layers.
• Leftover error. Even the BatchTopK crosscoder's error terms still contain a lot of information about chat behavior, so the feature list is not the whole difference.
• New or shifted? The crosscoder cannot tell a truly novel feature from an old one whose activation pattern shifted. The authors call this an open challenge.
• Access and labels. You need both models' internals, and a feature label comes from top examples and can be wrong.
• Other diffing tools. Lindsey and colleagues note that a base-model SAE often transfers to the fine-tuned model, which gives another kind of diff. A 2025 paper (arXiv 2510.13900) used even simpler activation differences and found that narrow fine-tuning leaves clearly readable traces. It also warns that narrowly fine-tuned models may not be realistic stand-ins for studying chat-tuning.

For auditing, a diff gives a place to start: look at what training changed instead of searching the whole model. A sketch: train a crosscoder on both models, sort features by relative norm, run a check such as Latent Scaling on the extremes, read top examples, then change the suspect features and see whether behavior follows.

It also helps ask how a narrow training step spreads, a question raised in emergent misalignment. A clean diff does not prove a model is safe, and a strange feature does not prove it is dangerous. It narrows where to look.

Frequently asked questions

What is the difference between a crosscoder and a sparse autoencoder?

A sparse autoencoder reads one set of activations from one layer of one model. A crosscoder reads and writes several, such as two models, and learns one shared feature list with a separate decoder direction for each.

Is a feature that appears only in the chat model proof of a new behavior?

No. Minder and colleagues showed that the L1 crosscoder can label a shared concept as chat-only. You need checks such as Latent Scaling, then examples and causal tests.

Can crosscoders compare any two models?

The original note says crosscoders can also be used across independent models with different architectures, but the main results are for a base model and its fine-tune. You need access to the internal activations of both.

Where does this sit in an alignment course?

It sits within interpretability and auditing. Learn AI Alignment Theory has an Interpretability course among its Advanced courses.

Get started

Learn AI Alignment Theory has 22 courses in three levels, from Basic to Advanced, with lessons of about 8 minutes, questions and hands-on activities. Lessons separate what is known from what is still open, and list their sources. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.