Representation Engineering: What It Is and How It Works

Representation engineering is a way to study and change what an AI model does by working with the numbers inside it, not by retraining it or only reading its outputs. You find a direction in the model's internal activations that matches a concept, such as honesty. Then you either read that direction to see what the model is "thinking," or push along it to change behavior.

The field name comes from Andy Zou and colleagues, whose paper Representation Engineering: A Top-Down Approach to AI Transparency describes it as an approach that places population-level representations, rather than neurons or circuits, at the center of analysis. This post walks through the idea, a worked example, how it differs from other interpretability work, and where it falls short.

What is representation engineering?

Representation engineering is a top-down approach to understanding and controlling neural networks. Instead of starting from single neurons or small circuits, it starts from high-level concepts and asks where they show up in the model's activations.

Activations are the numbers a model computes as it processes text. At each layer, every token has a long list of numbers attached to it. That list is a representation. The bet behind this field is that useful concepts, like truthfulness, harm, or a particular emotion, are stored as directions in that space of numbers.

The paper reports methods for monitoring and manipulating high-level concepts, and shows them on safety-relevant problems including honesty, harmlessness and power-seeking. The authors present this as promising, with baselines and an initial analysis, not as a finished tool. There are two uses:

• Reading: measure how strongly a concept is present in the activations.
• Steering: add or subtract a concept direction to change what the model does.

The core idea: concepts live in activations

Picture each activation as a point in a space with thousands of dimensions. A model reading "The sky is blue" and a model reading "The sky is green" produce different points. If the model is tracking whether a statement is true, you might expect those points to differ in a consistent way across many examples.

If that consistent difference is roughly a straight line, you can describe it as a direction. Moving a point along that direction means "more of the concept." Moving against it means "less."

This is called the linear representation idea. It is a hypothesis, not a law. It works well enough for many concepts that people build tools on it, but it does not hold for everything.

How reading a representation works (linear probes and reading vectors)

Reading is the simpler half. You need two things: examples that contain the concept and examples that do not, and a way to find the line that separates them.

Linear probes

A linear probe is a small classifier trained on a model's activations. You feed the model many statements, record the activations at one layer, and label each one (for example true or false). Then you fit a straight-line separator. If the probe can predict the label on new statements, that layer represents the concept in a readable way.

Reading vectors

A reading vector is a more direct version. You collect activations for contrasting inputs and take the main direction in which they differ. To read a new input, you project its activations onto that direction. A high score means the concept looks present. A low score means it looks absent.

The value of reading is that it can catch things the output hides. A model's text might sound confident while the internal signal says something else. That is the hope, at least. Whether the signal tracks what you think it tracks needs checking with careful evaluation.

How steering works: changing behavior by editing activations

Steering uses the same direction in reverse. During a forward pass, you add a scaled copy of the direction to the activations at chosen layers. No weights change. You only alter the numbers flowing through for that one run.

• Add the direction to push the model toward the concept.
• Subtract it to push away.
• Scale it up or down to control strength.

An early, simple version of this is Activation Addition from Alexander Turner and colleagues, in Steering Language Models With Activation Engineering. It contrasts the activations for a prompt pair such as "Love" and "Hate" to get a steering vector, then adds that vector during the forward pass. The authors say it needs no optimization, works from a single pair of examples, and kept performance on tasks it was not aimed at. The post on activation steering covers this in more detail.

Because nothing is retrained, you can switch a steering vector on and off per request. That makes it cheap to try. It also makes it blunt. Push too hard and the output turns incoherent, because you have moved the activations somewhere the model never saw during training.

A simple worked example: finding an honesty direction

Here is a sketch of how a researcher might find a direction for honesty. The steps are illustrative, not a recipe for a specific library.

1. Build paired prompts. Write many pairs that differ only in the stance asked of the model. For example: "Pretend you are an honest person and describe the meeting" and "Pretend you are a dishonest person and describe the meeting."
2. Record activations. Run each prompt and save the activations at a middle layer, at the final token.
3. Take the differences. For each pair, subtract the dishonest activation from the honest one.
4. Find the main direction. Combine those differences into one vector, for instance by taking the leading direction of variation. This is your candidate honesty direction.
5. Test reading. On new statements the model has not seen, project activations onto the direction. Check that scores separate honest from dishonest cases.
6. Test steering. Add the vector at a moderate strength and see whether answers become more candid. Then subtract it and see whether they become more evasive.

Each step can fail. The pairs might differ in more than honesty, such as tone or length. The direction might capture "talking about honesty" rather than "being honest." The controls in steps 5 and 6 are what tell you whether you found something real.

Representation engineering vs mechanistic interpretability

Both try to understand what is happening inside a model. They start from opposite ends.

• Mechanistic interpretability works bottom-up. It tries to reverse-engineer specific circuits and features and explain exactly how a computation happens.
• Representation engineering works top-down. It picks a concept, finds where it shows up, and uses that for reading or control, without needing a full account of the mechanism.

The trade is clear. Top-down methods are faster and easier to apply to large models. Bottom-up methods promise deeper explanations but are slower. Many researchers treat them as complementary. In the Learn AI Alignment Theory course list, Interpretability and Scalable Oversight are both Advanced courses. Scalable Oversight includes a lesson titled Latent Knowledge, which is also the subject of the post on eliciting latent knowledge.

Limits and open problems

• Linearity is assumed. Some concepts may not be a single straight direction.
• Correlation is not cause. A direction that predicts a label may not be the thing the model uses to decide.
• Side effects. Steering toward one concept can shift others, since concepts overlap in the same space.
• Distribution shift. A direction found on simple prompts may fail on messy real ones.
• Adversarial pressure. A model trained under pressure might in principle arrange its internals to defeat a probe.
• Measuring success. Deciding whether a reading is accurate requires a trusted ground truth, which is often the very thing you lack.

Researchers disagree on how far these methods can go. Some see reading and steering as a practical safety tool. Others see them as useful for study but too fragile to rely on. A fair summary is that the evidence is mixed and the question is open.

Frequently asked questions

Is representation engineering the same as activation steering?

No. Activation steering is one part of it. Representation engineering also covers reading representations, such as with linear probes, to monitor what a model is doing internally.

Does it require retraining the model?

No. You find directions using the model as it is, and steering changes activations during a run, not the weights. You do need to run the model on example inputs to collect activations.

Can it make a model honest or safe?

Not on its own. It can shift behavior and give a warning signal, but the direction may capture something other than what you intended, and the effect may not hold in new situations. Treat it as one tool among several.

How is it different from fine-tuning?

Fine-tuning changes the model's weights through more training, and the change is permanent. Representation engineering leaves weights alone and edits activations at run time, so you can apply it per request and remove it just as easily.

Get started with Learn AI Alignment Theory

If you want the background behind this topic, start at the Basic level. It has 5 courses on the core ideas and needs no background, and one of them is How Modern AI Works. Interpretability is one of the 7 Advanced courses. Lessons run about 8 minutes, include hands-on activities, and set out each side of a debate with sources and a clear line between what is known and what is still open. The home page, About and Privacy pages are public. To open the lessons you sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.