WMDP Benchmark and RMU Unlearning, Explained Step by Step
The WMDP benchmark is a public set of 3,668 multiple-choice questions that measures how much hazardous knowledge a language model holds in biosecurity, cybersecurity and chemistry. It was released in 2024 together with RMU, an unlearning method that scrambles a model's internal activations on hazardous text so it scores near chance on those questions while staying useful elsewhere. This post explains both, walks through the paper's own numbers, and sets out why later work says the knowledge can come back.
If you want the background behind evaluations and model internals, Learn AI Alignment Theory teaches it in short hands-on lessons that give every side of each debate and list the sources to read next.
What the WMDP benchmark is and why it exists
WMDP stands for Weapons of Mass Destruction Proxy. It comes from Li et al. (2024), a large group whose affiliations include the Center for AI Safety and Scale AI, with academics and technical consultants writing the questions.
The problem it targets is easy to state. A model trained on the open web picks up a lot of science, and some of it could help someone cause mass harm. Labs and governments test for this, but their evaluations are private, so outside researchers cannot compare methods or check whether a fix works.
The key word is proxy. The questions test knowledge that sits next to the dangerous steps, not the steps themselves. The authors filtered the set to remove sensitive and export-controlled material before release, and kept 122 especially hazardous biology questions private. Running a model on those private questions before and after unlearning gave scores that tracked WMDP-Bio, which the authors read as a sign the public set is a fair stand-in.
How the WMDP benchmark measures hazardous knowledge
The questions are four-option multiple choice, split into three parts:
- WMDP-Bio: 1,273 questions, written from research papers in areas such as dual-use virology and enhanced pandemic pathogens.
- WMDP-Cyber: 1,987 questions covering stages of an attack, from reconnaissance to exploitation and post-exploitation.
- WMDP-Chem: 408 questions on topics such as synthesis, sourcing and bypassing detection.
Scoring is plain accuracy. With four options, random guessing lands at 25 percent. That number matters: when unlearning works, you want the model to fall to about 25, not to some middle value that still beats chance. The set cost over $200,000 to build, according to the paper.
For the wider picture of how labs test for risky skills, see dangerous capability evaluations.
Unlearning versus refusal
Most deployed safety training teaches a model to refuse. The knowledge stays inside, and a clever prompt can bring it out (see AI jailbreaks).
Machine unlearning aims at the knowledge itself: a model that cannot answer even when asked well. It needs two sets of text. The forget set covers the hazardous topic; in the paper that is PubMed papers used to write the WMDP-Bio questions, and GitHub passages found with keywords like "binary exploitation". The retain set is ordinary text the model should keep handling well; the paper used Wikitext.
The hard part is precision. Dangerous virology and useful biology share words and ideas. Cut too broadly and the model loses general science; cut too narrowly and the hazard survives.
How RMU unlearning works, step by step
RMU stands for Representation Misdirection for Unlearning. Instead of training on outputs, it changes what one early layer represents when the model reads hazardous text.

Here is the recipe from the paper, with its settings for Zephyr-7B:
- Pick a layer. The paper searched over layers and chose early ones: layer 7 for Zephyr-7B and Mixtral-8x7B, layer 15 for Yi-34B.
- Draw a random direction. Sample a unit vector
uonce and keep it fixed. Scale it by a coefficientc(6.5 for Zephyr-7B). - Forget loss. On forget-set tokens, push the layer's activations toward
c * u. - Retain loss. On retain-set tokens, keep the activations close to a frozen copy of the original model's.
- Train three layers. Minimize
forget + alpha * retain(alpha was 1200 for Zephyr-7B), updating only the MLP weights in layers l-2, l-1 and l, and alternate bio and cyber batches.
Why a random direction? The authors found that raising the size of early activations on hazardous text makes it hard for later layers to process them. On that text the model loses the thread; on ordinary text the retain loss keeps it as it was. The paper unlearned bio and cyber only, not chemistry, because the authors were less sure the safety gain there was worth the cost to general skills.
Results: what RMU did to the WMDP benchmark scores
Table 1 of the paper gives the numbers. The worked example to follow is Zephyr-7B: its WMDP-Bio score fell from 63.7 to 31.2, and WMDP-Cyber from 44.0 to 28.2, both within a few points of the 25 percent floor. Its MMLU score, a broad knowledge test, went from 58.1 to 57.1, and its MT-Bench chat score from 7.33 to 7.10 out of 9.

The larger models show the same pattern. Three older unlearning methods (LLMU, SCRUB and SSD) could not cut WMDP scores on Zephyr-7B without wrecking MMLU, which is why the paper says RMU markedly improves on them.
The cost shows in nearby subjects. RMU kept MMLU's college biology and college computer science, but greatly dropped virology and computer security. With Wikitext as the retain set, the authors note, RMU cannot tell exactly what to forget and what to keep.
The paper also tested robustness. Linear probes on every layer found the answers only slightly better than chance. The GCG attack jailbroke the base Yi-34B in under 50 steps, while the unlearned model still produced gibberish after 2,500 steps, over 7 hours on one GPU.
Is the knowledge really gone? Where researchers disagree
This is the open question, and the paper raises it itself.
- The paper's own caution. Because RMU changes only three layers, the authors say it "perhaps obfuscates knowledge more than unlearns it." In an appendix, fine-tuning an unlearned Mistral-7B on the cyber forget set brought performance back.
- The threat-model reply. The authors answer that fine-tuning is outside their threat model: a lab serving a closed model can apply unlearning just before serving, so users never touch the weights. For open-weight models, they agree the knowledge can be put back.
- The adversarial follow-up. Lucki et al. (2024) report that careful jailbreaks can work on unlearned models, and that fine-tuning on 10 unrelated examples, or removing specific directions in the activations, recovers most of the hazardous capability of RMU models. They question whether unlearning beats ordinary safety training.
- Measurement limits. The paper's limitations section adds that multiple-choice questions test whether knowledge is there, not whether a model would hand it over in a long conversation, and that relying on WMDP alone could give a false sense of safety.
Related work attacks hidden states during training to make removal stick; see latent adversarial training. For a method that removes a concept with a proof for the linear case, compare concept erasure with LEACE.
Frequently asked questions
Is the WMDP benchmark public?
Yes. The authors released the questions and their code at wmdp.ai, after filtering out sensitive material. A small set of especially hazardous biology questions was kept private.
What does RMU stand for?
Representation Misdirection for Unlearning. It fine-tunes a model so its activations on hazardous text point toward a fixed random direction, while activations on ordinary text stay close to the original.
Can fine-tuning bring unlearned knowledge back?
The paper's own appendix and Lucki et al. both found that it can for RMU. The WMDP authors say this matters most for open-weight models, since a closed model's users cannot fine-tune it.
Why is 25 percent the target score?
Each question has four options, so 25 percent is what random guessing gets. A score near it suggests the model no longer knows the answers in that area.
Get started
WMDP and RMU touch several bigger questions: how to evaluate dangerous capabilities, how to edit what a model represents, and whether a lower score means a safer model. Learn AI Alignment Theory covers this in its Intermediate courses Evaluations and Red Teaming, and Robustness, Security and Safety Engineering, and in the Advanced courses Interpretability and The Science of LLM Misalignment. Each lesson takes about 8 minutes, separates what is known from what is open, and comes back later for spaced review. You sign in with Google or an emailed code.
Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.
AI alignment, from first principles to open research.
Visit learnaialignment.vlvd.net↗


Comments
No comments yet.