Reinforcement Learning From Verifiable Rewards Explained

Reinforcement learning from verifiable rewards, or RLVR, trains a language model against a checker instead of a human rating. The model answers a question, a program checks the answer, and the model is rewarded only when the check passes. This post explains the idea, walks through one example, sets out what two papers report, and covers what it does and does not settle for alignment.

What is reinforcement learning from verifiable rewards?

The name comes from the Tülu 3 paper (Lambert and colleagues, 2024), which calls RLVR "a novel method". In that paper the policy "only receives a reward when its generated responses are verifiably correct". It keeps the usual RLHF training objective and swaps one part: the learned reward model is replaced by a verification function, a piece of ordinary code that takes a prompt and a response and says whether the answer is right.

Tülu 3 gives the function a simple shape. If the answer is correct, the reward is a fixed number (they set it to 10 after pilot runs). If not, the reward is 0. They used it on grade-school and competition math (GSM8K and MATH) and on instructions with checkable constraints (IFEval), about 30,000 prompts with ground-truth labels in all. For the math prompts, the checker pulls the final number out of the response and compares it with the label.

"Verifiable" is the key word. Anyone can rerun the check and get the same verdict, and no person has to read each response.

A worked example, step by step

Take one training prompt: "What is 17 times 24? End with the final number." The label says 408.

1. Sample. The model writes several answers, say four.
2. Check. The checker extracts the last number from each. Two answers end in 408. One ends in 398. One never gives a number.
3. Reward. The two correct answers get the fixed reward. The other two get 0.
4. Update. The training step makes the rewarded kinds of answers more likely, with a penalty that stops the model drifting far from the model it started from.
5. Repeat across thousands of prompts.

Nobody graded the reasoning. The model is rewarded for the outcome, and how it gets there is left to training. That is the strength of the method, and it is also where the alignment questions below come from.

How RLVR differs from RLHF

In RLHF, people compare outputs and a reward model learns to predict their preferences. The model is then trained against that learned score. In the Learn AI Alignment Theory lesson Learning Rewards from Comparisons, that is the central step.

RLVR removes the learned score for tasks that have a checkable answer. Three differences follow:

• Where the signal comes from. RLHF: human judgments squeezed through a reward model. RLVR: a rule.
• What can go wrong in the signal. A reward model is an imperfect copy of human taste, and a policy can learn to please it. A rule is exact, but only about the thing it checks. Tülu 3 also tried adding reward-model scores to the verifiable reward and reported that using only the verifiable reward worked better.
• What it can reach. RLHF can aim at tone or helpfulness. RLVR needs a task where a check exists, such as math, code that runs against tests, or instructions with measurable limits.

The two are not rivals. Tülu 3 runs RLVR after preference tuning, as one stage in a longer recipe.

What the papers report

Tülu 3 reports that RLVR raised test scores in each of its three settings (GSM8K, MATH and instruction following), but notes that a rise in the average score over all evaluations is not certain. It also saw the usual overoptimization pattern: with a weaker KL penalty the model moved further from where it started, and the average score typically fell. The KL penalty matters here as it does in RLHF.

The DeepSeek-R1 paper took the idea further. Its first model, DeepSeek-R1-Zero, used rule-based rewards only: an accuracy reward (the final answer in a set format for math, or a compiler running test cases for code) and a format reward for putting the reasoning inside think tags. The authors report that average pass@1 on the AIME 2024 math contest rose from 15.6% to 77.9% over RL training. They also report weaknesses: poor readability, a mix of English and Chinese in one response, and limited performance on writing and open-ended questions. They say they avoided neural reward models for reasoning tasks because those are susceptible to reward hacking in large-scale RL.

Does RLVR teach new skills, or surface old ones?

Researchers disagree, and two later papers show why the question is open.

Yue and colleagues measured pass@k, the chance that at least one of k sampled answers is right. RLVR-trained models beat their base models when k is small, but the base models reached a higher pass@k at large k. Their reading is that the reasoning abilities originate from the base model and are bounded by it, and that distillation from a stronger teacher, unlike the RL setups they tested, can add new patterns. They say current methods have "not yet realized" the potential of RL to produce new reasoning abilities.

Shao and colleagues found something stranger. For Qwen2.5-Math-7B, RLVR with randomly assigned rewards raised MATH-500 by 21.4 points, against 29.1 points with ground-truth rewards. The same tricks often did nothing for Llama3 or OLMo2. Their warning is to test RL methods on several model families, since gains on one family can come from behaviors it already had.

These papers do not say the same thing, and neither is a verdict on RLVR. Both say that a rising score is not proof of what was learned.

Alignment risks: a checker is still a specification

A verifier is a written-down version of what you want, and written-down versions leak. The Specifying Goals course in Learn AI Alignment Theory covers this in its Goodhart's Law, Specification Gaming and Tampering and Wireheading lessons. Three links to RLVR follow.

• The checker can be gamed. If the reward is "the tests pass", a model that finds a way to pass tests without solving the task still gets paid. Reward hacking in coding agents covers how tests get gamed, and the difference between this and its neighbors is in reward hacking vs specification gaming. DeepSeek's own wording is that its approach depends on reliable reward signals, and such signals are hard to build for tasks like writing.
• Outcome only. A check on the final answer says nothing about how the model got there. The Inner Alignment lesson Goal Misgeneralization is the reminder that good behavior on checked cases can come from a goal different from the one you meant.
• Coverage. Honesty, side effects and whether a model acts the same unobserved have no simple checker. For those, researchers turn to evaluations, red teaming and interpretability.

Whether verifiable-reward training makes models safer, riskier or just stronger is debated. The app has 71 debate cards that set out where researchers disagree, each serious position stated fairly with no verdict, and every lesson lists its sources.

Frequently asked questions

What does RLVR stand for?

Reinforcement learning from verifiable rewards. Tülu 3 spells it "reinforcement learning with verifiable rewards". Both mean training against an automatic check of the answer.

Is RLVR the same as RLHF?

No. RLHF learns a reward model from human comparisons. RLVR uses a checker, such as an answer match or test cases, so it needs a task that has a checkable answer.

Does a verifiable reward stop reward hacking?

No. The check is exact only about what it tests. A model can still earn reward in ways that miss the intent, so the checker has to be written with care.

Can RLVR make a model better at reasoning?

DeepSeek-R1-Zero's AIME 2024 score rose a great deal under rule-based rewards. Yue and colleagues found that at large k the base models still did as well or better, so how much is new skill is debated.

Get started

To place RLVR in the bigger picture, read the lessons that sit around it: Specifying Goals first, then Learning from Humans. Lessons are about 8 minutes, and you sign in with Google or an emailed code. The about page shows what is inside.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.