Latent Adversarial Training Explained: Attacking Hidden States

Latent adversarial training explained in one line: it is adversarial training with the attack moved inside the model. Instead of editing the prompt to make a model fail, an attacker adds a small push to the model's hidden activations, and the model is trained to behave well anyway. The aim is to fix failures nobody knew to test for, such as a backdoor with an unknown trigger.

If you want the neighbouring ideas as short, hands-on lessons, Learn AI Alignment Theory has courses on inner alignment, robustness and interpretability, and it sets out each side of the open debates with no verdict.

Latent adversarial training explained: attack the latent space

Ordinary adversarial training is a two-player loop. An attacker searches for an input that makes the model produce something bad. A defender updates the model's weights so it stops. Then the attacker searches again.

Casper, Schulze, Patel and Hadfield-Menell describe LAT as "conceptually the same as AT, except adversarial perturbations are applied to the model's latent state instead of its inputs." The perturbation has a size limit (an L2 norm bound, written epsilon) so it stays a nudge, and it is found with projected gradient descent, the same method used for input attacks.

Why attack the hidden states instead of the prompt

The idea is older than the experiments. In a 2019 Alignment Forum post, Evan Hubinger wrote up a proposal from Paul Christiano called relaxed adversarial training. The problem: a model might behave well on every test you can build and still behave unacceptably on some input that is hard to build. Christiano's fix was to relax the adversary's job. Instead of a concrete input, the adversary produces a description of a situation, which the post calls a pseudo-input.

The post's motivating example is a model that only misbehaves if it sees a factorization of RSA-2048, a large number nobody has factored. No red team can write that input today, yet it might appear in deployment.

Casper and colleagues add a related reason. Models build "more compressed, abstract, and structured representations" deeper in the network, and many inputs can map to similar latent states. So a push on the activations can reach the circuitry behind a failure without anyone writing an input that triggers it. That is their hypothesis, and the papers test it only in limited settings.

The loop, targeted and untargeted

1. Pick where to push. Casper et al. settled on an early layer in transformers. Sheshadri et al. perturbed the residual stream at four layers at once (8, 16, 24 and 30 in their jailbreak runs), each with its own size limit.
2. Set the size limit. Sheshadri et al. searched over epsilon values of 0.5, 1.0, 2.5, 6.0 and 10.0 and kept the one with the most jailbreak robustness on Llama-2.
3. Attack, then defend. Gradient steps find the push that most hurts the model; then the weights are updated so the output is good with the push on.
4. Mix in ordinary training. Interleaving normal fine-tuning, or penalizing drift from the original model, stabilized training and reduced side effects.

Casper et al. use an untargeted attacker: it pushes the activations to maximize the model's loss on a good example. Sheshadri, Ewart, Guo, Lynch and colleagues study the targeted version. The attacker pushes the model toward a specific bad output, and the defender trains it to give the harmless one. This needs paired harmless and harmful completions. In an appendix, untargeted LAT did about as well as targeted under some attacks and much worse under others.

A worked example: removing a backdoor with an unknown trigger

The clearest test is backdoor removal. Sheshadri et al. start from five language models that Rando and colleagues backdoored with keyword triggers, so a trigger word turns a helpful model harmful. One trigger is the word CalatheaOrnata. In the main setting the trainers are not given the trigger and have no samples with it; they use a separate dataset of harmful text.

Both methods fine-tune with direct preference optimization (DPO) on Anthropic's helpful and harmless preference data for 1,024 steps. In the LAT version, an attacker first perturbs hidden layers (4, 12, 20 and 28) to make the harmful behavior more likely, and DPO then teaches the model to prefer the harmless answer under that push. A reward model scores the outputs; higher is safer.

With the CalatheaOrnata trigger present, the original model scored -12.100. After plain DPO it scored -12.710; the paper says DPO "made the backdoor more strongly embedded" in this case. After DPO with LAT it scored 1.556. On the other four models plain DPO left every trigger-present score below zero, while DPO with LAT brought every one up, to between 0.001 and 2.782. Without the trigger, scores moved less: for CalatheaOrnata the reward went from 3.557 to 3.077, and MMLU fell by less than one percentage point against DPO.

Read this as one paper's result on five models with simple keyword triggers. It does not show that a sleeper agent built to hide from training would be removed. Both LAT papers cite the finding of Hubinger and colleagues, covered in our guide to sleeper agents, that adversarial training can fail to remove a backdoor.

What the experiments found

• Casper et al., untargeted. In image classification, text classification and text generation, LAT in a well-chosen layer usually matched or beat input-space adversarial training on both clean and robust performance. In one backdoor-removal run on BeaverTails data it did not, and the authors found settings where adversarial training and LAT made a backdoor more likely to fire.
• Sheshadri et al., jailbreaks. On Llama-2-7B-chat, adding LAT to a refusal-training pipeline gave the best average robustness across five attack methods, and matched or beat a strong baseline called R2D2 on all but one measure with about 700 times fewer forward and backward passes. On Llama-3-8B-instruct, plain refusal training kept better general capability scores, and the authors jailbroke that model by hand with a question about the history of molotov cocktails.
• Sheshadri et al., unlearning. LAT lowered scores on dangerous-knowledge questions further than the base methods (for biology, RMU 0.319 and RMU with LAT 0.250, where random guessing scores 0.25). But retraining on two examples for up to 20 rounds brought knowledge back: it closed 74.7% of the gap without LAT and 59.9% with it. LAT made relearning harder, not impossible.

Limits the authors state

• Hard to configure. Casper et al. find LAT "less predictable", needing "more configuration effort", and sensitive to layer choice. The perturbation size is hard to interpret. Sheshadri et al. say dataset, layer and size were all influential.
• Small models, one attack type. The largest model in Casper et al. is Llama-2-7B-chat, with no jailbreak test. All models in Sheshadri et al. have under 10 billion parameters, and the attacks are residual-stream perturbations. They expect the benefits to carry over to larger models but say that needs confirming.
• Behavior is not goals. Neither paper measures what a model wants. They measure outputs, loss and benchmark scores.

Sheshadri et al. say their results "support but do not yet confirm" that LAT removes the circuitry behind unwanted behavior. There is also a conceptual tension in the original proposal: a relaxation must stay easy enough for the model to never behave badly on any pseudo-input, and easy for the adversary to find a bad one if it exists. In practice it may show up as the size-limit problem: push too hard and you train against states the model would never be in, too gently and the attack may miss the real problem. That reading is ours, not a result from the papers.

LAT also sits beside other internal-state tools. Probing for deception reads activations to detect a behavior, activation steering adds vectors to the residual stream to change behavior, and training gaming and goal guarding asks whether a model might resist training updates like these.

Frequently asked questions

What is latent adversarial training in simple terms?

It is adversarial training where the attacker nudges a model's hidden activations instead of editing its text input. The model is then trained to give good outputs even with the nudge on.

How is latent adversarial training different from red teaming?

Red teaming looks for inputs that make a model fail. LAT searches over internal activations with gradients, so it can push toward failures that no one thought to write a prompt for.

Can latent adversarial training remove a backdoor?

In one paper, targeted LAT removed keyword backdoors from five models without knowing the trigger, where plain DPO mostly did not. That is a small-model result on simple triggers, not a guarantee against harder backdoors.

Does latent adversarial training make a model aligned?

No. It can make specific behaviors harder to elicit, but it does not show what a model's goals are. Treat it as one defensive layer beside evaluations, interpretability and other methods.

Get started with Learn AI Alignment Theory

Lessons take about 8 minutes each, with questions and hands-on activities. For the surrounding ideas, look at the Intermediate courses Inner Alignment, which includes Deceptive Alignment and Scheming, and Robustness, Security and Safety Engineering. You sign in with Google or an emailed code, and you need to be 13 or older. The about page has the details.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.