How Human Preference Data Is Collected for RLHF

If you want to know how human preference data is collected for RLHF, start with one idea: the model never gets a grade for being "good". It gets many small judgments of the form "this answer is better than that one". This post follows one prompt through the process, using what two labs have published about their own data collection, and then looks at what the method cannot tell you.

What preference data is, and how human preference data is collected for RLHF

RLHF means reinforcement learning from human feedback. "Be helpful and honest" is hard to write as a formula, but a person can look at two answers and say which is better. So teams collect those choices, train a reward model to predict them, and tune the main model against that reward model. The summarization paper by Stiennon and colleagues describes this recipe: collect a dataset of human comparisons between summaries, train a reward model to predict which one people prefer, then fine-tune the summarizer with reinforcement learning.

One record of preference data has three parts: a prompt, two or more responses, and a label saying which response the person preferred. The rest of the pipeline is about producing those records. For the math that turns a label into a training signal, read the Bradley-Terry model in RLHF explained simply. For the whole loop, read how RLHF works.

Step 1: where the prompts come from

The InstructGPT paper from OpenAI (Ouyang and colleagues, 2022) names two sources. The first is prompts submitted through the OpenAI API. The second is prompts the labelers wrote themselves. The paper says it needed these to bootstrap the process, because instruction-like prompts were not common on the API at first. Labelers wrote three kinds: plain (any task, with enough variety), few-shot (an instruction plus several example query and response pairs) and user-based (prompts matching use cases that people had listed in API waitlist applications).

The paper reports about 33k training prompts in the dataset used for the reward model. It also says the data is over 96% English. That one figure already shows a design choice: the prompts decide which behavior can be rewarded at all.

Step 2: showing people responses and asking for a ranking

For each prompt, the model writes several answers. In InstructGPT, labelers saw between K = 4 and K = 9 responses and ranked them. A ranking of K answers gives K choose 2 pairwise comparisons, so 4 answers give 6 comparisons and 9 answers give 36. The authors say they did this to speed up comparison collection.

Anthropic's team used a different format in its helpful and harmless assistant paper. Crowdworkers chatted with a model. At each turn they saw two possible responses and chose one, and the conversation continued from the chosen one. Workers on the helpfulness task were told to choose the more helpful and honest response. Workers on the red-teaming task, who tried to provoke harmful behavior, were told to choose the more harmful one. The two goals gave two separate datasets.

A worked example, invented for illustration. The prompt is "Write a short note to my neighbor about their barking dog". The model writes four answers. A labeler ranks them: the polite note with one concrete request first, a polite but vague note second, a blunt note third, an off-topic reply last. Those four rankings produce six pairs, and each pair is one training example for the reward model.

Step 3: who the labelers are and how they are guided

The two papers made different choices. OpenAI hired about 40 contractors through Upwork and ScaleAI. It ran a screening test meant to find people who were sensitive to the preferences of different demographic groups and good at spotting potentially harmful outputs. It then trained them with an onboarding process, wrote detailed instructions for each task, and answered questions in a shared chat room.

Anthropic invited master-qualified, US-based Mechanical Turk workers. It found the roughly 20 most prolific workers, who together gave about 80% of the data, and kept working with a select group. It did not filter them by agreement on the choices. It spot-checked how well they wrote and how varied their conversations were, on the guess that people who write well also judge well. Otherwise it let them use their own intuitions for "helpful" and "harmful", hoping that data diversity and the "wisdom of the crowd" would pay off.

Neither choice is neutral. A detailed instruction sheet and an open-ended one both teach the reward model what "better" means.

Step 4: checking whether the labels agree

People disagree on close pairs. InstructGPT measured it: training labelers agreed with each other 72.6 percent of the time (plus or minus 1.5), and a separate group of held-out labelers 77.3 percent (plus or minus 1.3). The paper compares this to 73 percent (plus or minus 4) for researchers in the earlier summarization work. The authors call these rates "quite high" given how complex the task is, and they also report that the labelers disagree on many examples.

Read the number both ways. About one comparison in four was settled differently by a second labeler. Yet each pair still carries a signal, and a reward model trained on many pairs can average over the noise. The same paper adds a cost point: most comparisons were labeled by one contractor, so the data cannot show where labelers split.

What the data cannot tell you

The InstructGPT authors state the limits plainly. The model is aligned to the preferences of its training labelers and, through the instructions, of the researchers. The labelers are mostly English-speaking people living in the United States or Southeast Asia, and the paper says this group is "clearly not representative" of everyone who will use the model. It also notes that labelers are not the users who wrote the prompts, so what a labeler thinks was intended can differ from what the user meant.

Two wider worries follow. First, a reward model is a stand-in for what people want, and pushing hard on a stand-in can go wrong: see reward model overoptimization. Second, people can only judge what they can check. The scalable oversight post covers ideas for that gap. Some labs also use AI feedback to produce part of the comparisons: see RLAIF and Constitutional AI. Researchers disagree about how far that can replace human labels, and no verdict is given here.

Frequently asked questions

What is human preference data in RLHF?

It is a set of records, each with a prompt, two or more model responses and a human label for which response was better. A reward model learns from the labels to predict human choices.

Why do people rank or compare answers instead of scoring them?

The papers above collect comparisons and rankings, and the summarization paper trains its reward model on comparisons between summaries. A ranking of K answers also gives many pairs from one task.

How many answers does a labeler see at once?

It depends on the project. InstructGPT showed between 4 and 9 responses to rank, while Anthropic's chat interface showed two at each turn.

Do labelers agree with each other?

Often, but not always. InstructGPT reported 72.6 percent agreement between training labelers and 77.3 percent for held-out labelers.

Who decides what counts as a better answer?

The labelers, within the instructions the lab writes. InstructGPT's authors say the model is aligned to the preferences of its labelers and researchers, not to a broad idea of human values.

Get started

To practice this rather than only read about it, the Learning from Humans course on Learn AI Alignment Theory has lessons such as Learning Rewards from Comparisons and Constitutional AI and the Limits of Feedback, each about 8 minutes, with questions and hands-on activities, and each lesson lists its sources. You sign in with Google or an emailed code. You can also read about the project or check the privacy page first.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.