RLHF Explained: How Chatbots Learn to Be Helpful

RLHF explained in one line: reinforcement learning from human feedback is a training step where people rank a model's answers, and the model is then tuned to produce the kind of answers people rank higher. It is a big part of why a chatbot answers your question instead of just continuing your sentence. If you are building a product on AI, knowing what this step does and does not do helps you predict where the model will help and where it will let you down.

The How Modern AI Actually Works course on Age of AI Entrepreneur covers this in its second lesson, From Raw Model to Helpful Assistant, and a reader signs in with an email and an emailed code to open it. This post goes deeper on that one step.

Why a raw language model is not yet an assistant

A language model starts out trained to predict the next token of text. That skill is powerful, but it is not the same as doing what you asked. Ask a raw model a question and it may continue with more questions, because that is what text on the web often does next. Our post on what a foundation model is shows how one raw model sits underneath many products.

The InstructGPT paper puts the problem plainly. Its abstract says that making language models bigger does not inherently make them better at following a user's intent, and that large models can generate outputs that are untruthful, toxic or simply not helpful. The authors call such models not aligned with their users. The fix they describe is fine-tuning with human feedback.

RLHF explained step by step

The paper, which fine-tuned GPT-3 into a family it calls InstructGPT, describes three stages.

• Step 1: demonstrations. People write the answers they would like the model to give to a set of prompts. The model is fine-tuned on these examples with ordinary supervised learning.
• Step 2: rankings. The model produces several answers to the same prompt, and people rank them from best to worst. A separate model, called a reward model, is trained to predict which answer the people would prefer.
• Step 3: optimization. The model from step 1 is tuned further with reinforcement learning, using the reward model's score as the reward. Answers the reward model likes become more likely.

Notice what the humans actually supply: a modest number of written examples and a larger number of comparisons. They do not rate every answer the model will ever give. The reward model stands in for them after training, which is what makes the method affordable. The idea of learning from human preferences between pairs of outputs is older than chatbots. A 2017 paper on deep reinforcement learning from human preferences did it for game and robot tasks, giving feedback on less than one percent of the agent's interactions.

What RLHF changed in the InstructGPT results

The numbers in the paper are the best evidence for why labs adopted the method. According to the abstract, in human evaluations the 1.3 billion parameter InstructGPT model was preferred over the 175 billion parameter GPT-3, even with 100 times fewer parameters. It also shows improvements in truthfulness and fewer toxic outputs, with minimal performance regressions on public benchmarks. In the paper's own tests, InstructGPT gave truthful and informative answers about twice as often as GPT-3 on the TruthfulQA benchmark.

The takeaway for a builder is that training the model to follow instructions can matter more than making it bigger. The abstract is also honest about the limit: even so, InstructGPT still makes simple mistakes. That is the third lesson of the course, What AI Gets Wrong, and our post on AI hallucination examples shows what those mistakes look like in real cases.

Whose preferences does the model learn?

RLHF teaches a model what some group of people prefer, and the group matters. The paper says directly that its procedure aligns the model with the stated preferences of a specific group of people, mostly its 40 hired labelers and researchers, rather than with any broader notion of human values. Serious people disagree on what to do about that, and the sources show two positions.

• Human feedback is the anchor. The paper's labelers were chosen by their performance on a screening test, and their rankings define what counts as a good answer. The case for this is that real human judgment is the thing you want the model to match, and they can study and improve how labelers are chosen and guided.
• Written principles can do part of the job. The Constitutional AI paper trains an assistant with a list of rules instead of human labels for harmful outputs. A model compares its own samples against the principles, and those AI preferences train a preference model. The case for this is that it controls behavior more precisely with far fewer human labels, and the abstract notes that the only human oversight is the list of principles.

Neither paper settles the question for every use. Both show that someone chooses the standard, and that choice shapes the product you build on top.

A worked example: run a tiny preference test on your own prompts

You will not run RLHF yourself in a first product. You can borrow its core move, which is asking people to pick the better of two answers. Here is a small test you can finish in an afternoon. The inputs and numbers are invented for illustration.

1. Collect 10 real inputs. Take them from your own users or the kind of requests you expect, such as customer emails your product will answer.
2. Write two prompts. Prompt A is your current instruction. Prompt B changes one thing, such as asking for a shorter reply that names the order.
3. Run both on every input. Hide which prompt made which answer, and pick the better one for each input in a few words.
4. Tally and read the reasons. If B wins 6 of 10, that is a hint, not proof. The reasons matter more: in the picture below, B won on tone but invented a deadline once.
5. Fix and repeat. Keep B, add a rule against inventing deadlines, and run the same 10 inputs again.

This is the same pattern the Evals course teaches in its lesson Building an Eval Set and Judging Outputs. For a checklist version, see our post on how to test LLM prompts before release, and for scaling the judging with a model, LLM as a judge.

What RLHF does not fix

It helps to keep the limits in mind when you plan a product.

• It does not make answers true. It makes them preferred. A confident, fluent wrong answer can still be ranked well if the labeler cannot tell it is wrong.
• It reflects its labelers. A model tuned on one group's rankings may not match your customers' taste or language.
• It does not remove randomness. The same prompt can still give different answers, which our post on why ChatGPT gives different answers to the same prompt explains.

So test the model on your own examples before you rely on it, and keep a person in the loop where a mistake is costly.

Frequently asked questions

What does RLHF stand for?

Reinforcement learning from human feedback. The model is tuned using a reward signal that comes, indirectly, from people's rankings of its answers.

Is RLHF the same as fine-tuning?

RLHF is one kind of fine-tuning. Step 1 is ordinary supervised fine-tuning on written examples, and steps 2 and 3 add rankings and reinforcement learning on top.

Do I need RLHF to build an AI product?

No. Most early products call a model someone else already tuned. What you can do is test prompts the way the example above shows, using people's preferences on your own inputs.

Why do chatbots still make mistakes after RLHF?

Training raises the chance of preferred answers but does not check facts. The InstructGPT paper itself says its models still make simple mistakes.

Get started

Open Age of AI Entrepreneur, sign in with your email and an emailed code, and work through the How Modern AI Actually Works course. Each lesson takes about nine minutes and lists its sources, and the terms page says every lesson is free to anyone with an account.