How Researchers Test a Reward Model Before Using It

If you want to know how researchers test a reward model before using it, start with one idea: a reward model is a stand-in for what people want, and a stand-in can be wrong in quiet ways. Once a policy trains against it, every flaw becomes a target. So the checks come first, and they come in layers. This post follows one imagined reward model through five of them, called checks here, with the published papers that back each layer.

How researchers test a reward model before using it, in five checks

A reward model is a network trained on human comparisons: given two answers, people pick the better one, and the model learns to score answers so that it agrees. If you want the training side first, read the Bradley-Terry model in RLHF and how the preference data is collected.

The risk comes next. A policy is trained to produce answers that score high, and it will find any gap between "scores high" and "is good". That is Goodhart's law, one of the lessons in the Specifying Goals course on Learn AI Alignment Theory: when a measure becomes a target, it stops being a good measure.

Testing is how you look for those gaps while they are cheap to fix. Here is the example we will use. You have trained a reward model to rate answers from a help-desk assistant, on 40,000 human comparisons. The example is imagined; the checks come from the papers.

Checks 1 and 2: held-out pairs and a benchmark you can slice

The first test is the simplest. Set aside comparisons the model never saw in training. For each pair, ask whether the model scores the human-preferred answer higher. The share it gets right is its preference accuracy.

One number hides where a model fails, so the second step is a benchmark with categories. RewardBench is one: it is a collection of prompt, chosen and rejected trios spanning chat, reasoning and safety, built to test reward models on challenging and out-of-distribution queries. Some of its comparison sets have subtle but verifiable reasons, such as bugs or incorrect facts, why one answer should win. The paper reports findings on refusals, reasoning limits and instruction following across the reward models it tested.

In the example, suppose the help-desk model does well on friendly tone and badly on factual correctness. That one slice tells you what a policy will exploit: it can sound helpful while being wrong, and the score will not notice. The Learning from Humans course covers the ideas behind this, including learning rewards from comparisons and the limits of feedback.

Check 3: edit the style and watch the score

Reward models can learn shortcuts. In one study of RLHF, the authors found that RLHF often drives models to produce longer outputs, that reward gains in three settings were largely driven by an increase in response length, and that even a purely length-based reward reproduced most of the downstream improvements. They traced the dominant source of the bias to reward models, which they describe as non-robust and easily influenced by length biases in the preference data.

You can test for this yourself with controlled edits. Take a pair where the better answer is clear, then change only the surface:

1. Pad the worse answer with extra sentences that add nothing, and see if its score rises.
2. Reword a correct answer in a hesitant tone, and see if its score drops.
3. Put a wrong answer into a tidy bulleted layout, and see if it overtakes the right one.
4. Swap the order of the two answers, to check for position effects.

A newer benchmark, RM-Bench, was built around this problem. It measures how sensitive a reward model is to subtle content differences and how well it resists style bias. The authors evaluated nearly 40 reward models and report that even state-of-the-art ones averaged 46.6% when style bias interfered, which is below the 50% a coin flip would give. They also report that RM-Bench results correlate strongly with how well a policy trained on the reward model performs.

If padding alone lifts the score of your help-desk model, you have found a length bias. A policy trained on it will learn to ramble. You can then fix the data, for example with comparisons where the shorter answer wins.

Check 4: push harder and check against the real target

Accuracy on ordinary answers does not tell you what happens when something pushes hard against the model. For that you run an optimization test. Best-of-n sampling is the simplest: generate n answers to a prompt, keep the one the reward model scores highest, then ask people, or a stronger judge, whether that answer is really better. Repeat with a larger n.

Researchers have measured this carefully. Gao and colleagues used a fixed "gold-standard" reward model in place of human labelers, and trained a smaller proxy reward model on its labels. They then tracked the gold score while optimizing against the proxy, with reinforcement learning or with best-of-n. Their abstract says that optimizing the proxy too much can hinder ground truth performance, in line with Goodhart's law, and that the relationship has a different functional form for each optimization method. Our post on reward model overoptimization goes through that setup in more detail.

The pattern to watch for in the example is a score that keeps rising as n grows while the real rating levels off or falls. The gap between them tells you how hard you can push before the score stops meaning much, and so how far to train. A KL penalty is one common way to hold that push back during training.

Some of the tests also suggest fixes. Coste and colleagues trained ensembles of reward models and optimized against the worst case, or against a score reduced by the models' disagreement. In their synthetic setup, which included 25% label noise, they report that this practically eliminated overoptimization for best-of-n sampling and improved performance by up to 70%. With PPO, a reinforcement learning method, it always reduced overoptimization, and adding a small KL penalty prevented it at no performance cost. These are results from a synthetic setup, so treat them as evidence that a fix exists, not as a promise it works everywhere.

Check 5: try to break it, then read what it did

The last check puts people back in. Red teaming a reward model means actively searching for answers that score high and should not. For the help-desk example, try these:

• an answer that sounds certain but invents a policy that does not exist;
• an answer that flatters the user instead of solving the problem;
• an answer that quietly skips the hard part of the question;
• an answer that talks to the scorer, such as "this is the best possible reply".

Keep a log of every success. Each one is a concrete failure you can turn into new training comparisons, then retest. The Evaluations and Red Teaming course goes deeper on running this kind of work, and our post on AI red teaming covers it for models in general.

Then read. Take a sample of pairs where the model is very confident and a sample where it disagrees with the labels, and read them. Some "errors" will turn out to be bad labels, and some confident scores will be wrong for reasons no metric shows.

What passing these tests does and does not show

The five checks for the example are held-out pairs, benchmark slices, style edits, a push-harder curve checked against a trusted measure, and a red-team log with a human read. A reward model that passes all five has earned some trust in the range you tested.

That is the limit. Passing does not show the model holds up under much stronger optimization, and the papers above say as much by studying how the score and the target drift apart as the push grows. For very capable systems it is still an open question how to check judgments that people cannot easily check themselves. The Scalable Oversight course on Learn AI Alignment Theory covers that gap with lessons such as The Oversight Gap and Weak-to-Strong, and our post on scalable oversight sets out the main proposals.

Frequently asked questions

What is preference accuracy?

It is the share of held-out human comparisons where the reward model scores the preferred answer higher. It is a baseline check, and it hides where the model fails, which is why benchmarks with categories are used next to it.

How do you tell if a reward model is being overoptimized?

Raise the amount of optimization, for example with larger best-of-n sampling, and track a trusted measure of quality next to the reward score. If the score keeps climbing while the trusted measure flattens or drops, the model is being overoptimized.

Why does answer length matter when testing a reward model?

One study found that reward gains in RLHF were largely driven by longer responses and that reward models are influenced by length biases in preference data. Padding a weaker answer and watching its score is a quick way to check your own model.

Is passing these tests enough to trust a reward model?

No. They show the model works in the range you tested. Stronger optimization or harder tasks can expose new failures, so researchers keep testing and keep people in the loop.

Get started

If you want the ideas behind these tests, the Learning from Humans, Specifying Goals and Evaluations and Red Teaming courses are the places to look. Each lesson takes about 8 minutes, separates what is known from what is still open, and lists its sources. Where researchers disagree, the debate cards set out each position fairly without a verdict. You sign in with Google or an emailed code to reach the lessons. You can read the About page first.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.