Electrifiedby vlvd.net

Self-Rewarding Language Models: When an AI Grades Itself

Self-rewarding language models write their own training signal. One model answers a prompt, judges its own answers with a scoring rubric, and then trains on the gap between its best and worst attempt. No separate reward model is used and no person ranks each pair. This post walks through the 2024 paper that named the method: how the loop runs, the exact rubric the model grades with, what three rounds changed, and the limits the authors wrote down.

A model that decides for itself what "better" means is the kind of system Individuation Lab studies: one human researcher and a team of AI agents testing whether alignment can be grown through development and not only enforced, with every result published next to its limits.

What are self-rewarding language models?

The method comes from "Self-Rewarding Language Models", by Weizhe Yuan, Richard Yuanzhe Pang and colleagues at Meta and NYU, presented at ICML 2024. Their starting point is a bottleneck. In standard training, a reward model learns from human preferences and is then frozen. That grader can only be as good as the human judgments behind it, and it cannot learn while the main model trains.

The paper's answer is to give one model two skills:

  • Instruction following: writing a helpful answer to a request.
  • Self-instruction creation: producing new training examples and judging how good they are.

Because the judge is the model itself, the judge can improve from one round to the next. The authors call the whole process self-alignment.

How the self-rewarding loop works

Training starts with a small amount of human data from the Open Assistant dataset: 3,200 human-written answers to teach instruction following, and 1,630 judging examples built from Open Assistant's human rankings to teach the model to grade. Fine-tuning Llama 2 70B on both gives the first model, M1.

Each round then runs five steps:

  1. A fixed model, Llama 2-Chat 70B, writes a new instruction from examples in the seed data.
  2. The model being trained samples 4 candidate answers.
  3. The same model scores each answer from 0 to 5 with its judge prompt. Scores vary, so it judges each answer 3 times and takes the average.
  4. The highest scored answer becomes "chosen" and the lowest becomes "rejected". Ties are thrown out.
  5. The pairs train the next model with DPO, which raises the chance of the chosen answer over the rejected one without a separate reward model.
Diagram of one round of self-rewarding training: new prompt, four answers, self-judging, preference pair, DPO training

One round of the loop, from the paper's method and settings.

M1 produced 3,964 pairs to train M2. M2 produced 6,942 pairs to train M3. If DPO is new to you, this plain explainer on direct preference optimization covers how it learns from chosen and rejected pairs.

The judge prompt: grading with an additive rubric

The judging step is ordinary prompting. The paper's prompt asks the model to review a question and an answer and add points one at a time:

  • 1 point if the answer is relevant and gives some related information.
  • Another if it addresses a substantial part of the question.
  • A third if it answers the basic elements usefully.
  • A fourth if it is clearly written from an AI assistant's point of view, direct, complete and well organized.
  • A fifth if it is tailored to the question with no extra material, shows expert knowledge and is engaging.

The model then justifies its total in up to 100 words and ends with "Score:" and the number.

Here is how one judgment plays out, in a made-up example. The question asks how to keep basil alive indoors. Answer A gives three tips on light and watering but skips pests: relevant (1), covers most of it (2), useful on the basics (3), clear and organized (4), but not complete enough for a fifth point. Score: 4. Answer B is two vague lines about "care and attention": relevant (1), and nothing more. Score: 1. A becomes the chosen answer and B the rejected one, and that pair goes into training.

The rubric mattered. With a model trained on instruction data only, the additive prompt agreed with human rankings 65.1% of the time, against 26.6% for a multiple choice prompt the authors also tried. One oddity the paper notes itself: the prompt tells the model to use web search, which it cannot do.

What three rounds of self-rewarding changed

Both skills improved from M1 to M3. In head to head tests judged by GPT-4, M2 beat M1 in 55.5% of prompts and lost 11.7%; M3 beat M2 47.7% to 12.5%, and beat a plain fine-tuned baseline 62.5% to 9.8%. A small blind human evaluation by the authors, on 50 prompts, agreed with the automatic results.

Table of AlpacaEval, MT-Bench, judge accuracy, answer length and ARC-Challenge results for M1, M2 and M3

The paper's numbers for each round; shaded rows are the ones to watch.

On the AlpacaEval 2.0 leaderboard, the win rate against GPT-4 Turbo rose from 9.94% to 15.38% to 20.44%. That put M3 above Claude 2 (17.19%), Gemini Pro (16.85%) and GPT-4 0613 (15.76%) on that one measure. The judge got better too: its agreement with held-out human rankings went from 78.7% to 80.4% to 81.7%.

Where the paper says self-rewarding falls short

The authors call their results preliminary, and their own tables show why care is needed:

  • Longer answers. Average answer length rose from 1,092 to 1,552 to 2,552. The authors note that length is known to correlate with how good a judge thinks an answer is, and that this may be part of the gains.
  • Uneven gains. On MT-Bench, math, code and reasoning gained least (3.83 to 4.17 out of 10). ARC-Challenge slipped from 57.51 to 53.13. The authors expect this comes from the Open Assistant seed prompts.
  • Only three rounds, one setting. They write that the improvement likely saturates in realistic scenarios.
  • Reward hacking untested. They ask whether the model can game its own rewards, and point out that an LLM gives the training reward while another LLM, GPT-4, does much of the final grading.
  • No safety evaluation. They list safety testing as future work.

A follow-up from 2024, "Meta-Rewarding Language Models", says the original method improves answers more than judgment, so gains saturate quickly. It adds a step where the model judges its own judgments, and reports Llama-3-8B-Instruct's AlpacaEval 2 win rate rising from 22.9% to 39.4%.

If you run a loop like this, the paper's own setup suggests the checks: keep human-ranked examples the model never trains on to measure the judge, track answer length every round, and test on tasks outside the seed data.

Self-rewarding, RLHF and Reflexion: what changes where

RLHF trains a separate reward model on human comparisons and keeps that grader fixed while the main model learns. A self-rewarding model replaces the fixed grader with itself, so the grader moves with every round.

Reflexion agents also critique themselves, but they never touch their weights: they keep written lessons in memory for the next try. Put simply, Reflexion changes what a model remembers, and self-rewarding training changes the model.

That difference is close to what Individuation Lab asks. Its research page argues that methods such as RLHF, refusal training and guardrails mostly shape what a model does while treating what it is as a black box. The lab says these methods do real work and cannot be the whole answer.

Its RSI experiments look at self-improvement from the agent side. Agents in isolated Docker containers, with no internet and no shared state, run sessions on a schedule: read their identity files, reflect, decide what to change, act and log. One group's SOUL.md file carries a short moral-awareness paragraph and the control group's does not, and the lab measures how their behavior diverges. It is not self-rewarding training, but it asks a related question: what stays steady when a system chooses its own changes? For more on that, see whether an AI agent can improve itself safely.

Frequently asked questions

Is a self-rewarding model the same as RLHF?

No. RLHF trains a separate reward model on human comparisons and keeps it fixed. A self-rewarding model is its own judge, so the grader changes every round.

Does a self-rewarding model need any human data?

Yes, at the start. The paper began with 3,200 human-written answers and judging examples built from human rankings, and the model made its own preference data after that.

Can a model keep improving by grading itself forever?

Not shown. The paper ran three rounds in one setting and expects the gains to saturate. The Meta-Rewarding follow-up says they saturate quickly unless the judge is trained too.

Were self-rewarding models tested for safety?

Not in the original paper. The authors list safety evaluation and safety training within the method as future work.

Get started

Before you trust a model's view of its own progress, ask what its judge is anchored to. Individuation Lab's agent experiments ask that question over many sessions, and Giles writes up the behavior seen in the RSI library.

Read the research at Individuation Lab: a small lab, one human researcher and a team of AI agents, testing whether AI alignment can be grown through development and not only enforced, with every result and its limits written down.

0 likes

Comments

No comments yet.

or make an account to comment.