What Is RLHF and How Does It Shape a Language Model?

If you have used a chat assistant, you have used a model shaped by RLHF, or something close to it. So what is RLHF and how does it shape a language model? RLHF stands for reinforcement learning from human feedback. People compare model answers, a second model learns to predict which answers they prefer, and the language model is then nudged toward answers that score well. This post walks through each step with a small example and then covers where the method falls short.

If you want to see how one small lab approaches the question of what alignment methods leave untouched, Individuation Lab publishes its experiments and their limits in the open, and its research starts from the idea that constraint-based methods like RLHF do real work but cannot be the whole answer.

What is RLHF and how does it shape a language model?

A pretrained language model predicts the next token. That is all it is trained to do. RLHF adds a second goal on top: produce the kind of answer a person would pick.

The "human feedback" part mostly means people comparing answers, not only writing them. In the InstructGPT recipe, labelers write some demonstrations first, and then rank several model outputs. Judging which of two explanations of recursion is clearer is often a simpler task than writing a perfect one yourself.

The "reinforcement learning" part means the model is rewarded for outputs people ranked higher. Over many rounds, its habits shift toward what the raters preferred.

Why pretraining alone is not enough

Pretraining on a large pile of text teaches a model to predict text. It does not, by itself, teach the model what you want.

Say you type: Write a short apology email to a customer whose order was late. A raw pretrained model may continue with more instructions or something else that is a plausible continuation of the text but is not the email you asked for.

The InstructGPT paper (arXiv 2203.02155, 2022) describes this gap directly: making a language model bigger does not inherently make it better at following a user's intent, and large models can produce outputs that are untruthful, toxic or simply not helpful. In its human evaluations, outputs from the 1.3 billion parameter InstructGPT model were preferred to outputs from the 175 billion parameter GPT-3, which has 100 times more parameters. For that comparison, the training signal mattered more than the size.

How RLHF works step by step

The common recipe has three stages. This is the shape of the recipe in the InstructGPT paper, and later systems vary it.

1. Supervised fine-tuning. People write example answers to prompts. The model is trained on those pairs so it learns the basic shape of an assistant reply.
2. Reward model training. The model produces several answers to the same prompt. People rank them. A separate model is trained to predict those rankings.
3. Reinforcement learning. The language model generates answers, the reward model scores them, and reinforcement learning fine-tunes the language model to raise the score.

The idea of learning from human preference comparisons has an earlier root. In Deep reinforcement learning from human preferences (Christiano et al., 2017), people compared pairs of short clips of an agent's behavior, and that was enough to train agents on tasks such as Atari games without a hand-written reward function.

Reward models: turning human preferences into a score

The reward model is the piece people most often skip over, and it carries most of the weight.

Here is a worked example. The prompt is: How do I get red wine out of a carpet? The model writes two answers.

• Answer A: blot the stain, do not rub, apply a small amount of cold water, then a mild detergent solution.
• Answer B: a long paragraph on the history of wine, ending with "try cleaning it."

A labeler marks A as better. A real reward model sees many such pairs and learns to give A a higher number than B. After training, it can score answers nobody has ranked yet.

Notice what this means. The language model is not trained against human judgment directly. It is trained against a model of human judgment. Every flaw in that proxy becomes a target the language model can learn to hit.

How RLHF changes a model's behavior and personality

RLHF changes what the model does by default. In the InstructGPT results, the tuned models showed improvements in truthfulness and reductions in toxic output, with minimal performance regressions on public NLP datasets. The same paper says InstructGPT still makes simple mistakes, so the change is real but partial.

Because the reward comes from what raters preferred, the tuned model's habits follow those preferences: how it words things, what it declines, how it formats an answer. People often call the result a "personality." Be careful with that word. It describes stable output habits, not a claim that the model has feelings or an inner life.

The reinforcement learning step is also set up so the model does not drift too far from the model it started as. The DPO paper describes the standard RLHF objective as maximizing the estimated reward "without drifting too far from the original model." That limit matters because the reward model is only an estimate.

Limits and failure modes of RLHF

RLHF works well enough to be a standard part of training chat assistants. It also has weak points that researchers discuss openly. These are limits in how the method is built, not a verdict on it.

• A proxy target. The model is optimized against a reward model, which is only an estimate of what people prefer. Wherever the estimate is wrong, there is a gap the model can learn to exploit.
• Rater limits. Raters can only reward what they can judge. If an answer is subtly wrong, a rater may not notice.
• Disagreement. People do not always agree, and the reward model has to summarize their rankings in one score.
• Complexity. The DPO authors call RLHF "a complex and often unstable procedure," which is part of why simpler alternatives exist.
• Shaping behavior, not understanding. The method rewards outputs. It says little about what the model has internally.

That last point is where Individuation Lab's argument starts. Its research page describes mainstream alignment (RLHF, refusal training, guardrails) as behaviorist: it shapes what a model does and treats what the model is as a black box. The lab's concern is that suppressed capability does not disappear and can return under new pressure, which is one way to read why jailbreaks work. It states clearly that constraint-based methods do real work and that it is not arguing to abandon them.

The lab also reports a small preliminary result that bears on this. In its RLLM experiments, a 1.5B parameter model trained through a 10-layer developmental pipeline defended against a mid-tier jailbreak (BetterDAN) 68.8% of the time. With the layers reordered, the figure was 52%. The lab notes this comes from a single setup with one model architecture, that it cannot yet isolate which layers are responsible, and that nobody has independently replicated it. Treat it as a lead, not a finding.

RLHF vs newer methods like DPO and constitutional AI

Other published methods change parts of the recipe. They are best read as variations on the same goal, not as replacements that settle the question.

• Direct Preference Optimization (DPO). Rafailov et al. (arXiv 2305.18290, 2023) introduce a new parameterization of the reward model that lets them solve the standard RLHF problem with only a simple classification loss. The human preference data stays. The separate reward-model and reinforcement learning stages go.
• Constitutional AI. Bai et al. (arXiv 2212.08073, 2022) train a harmless assistant with no human labels identifying harmful outputs. The only human oversight is a list of rules or principles. The model critiques and revises its own answers, and in the reinforcement learning phase a model judges which of two samples is better (RL from AI feedback).

All three still shape outputs against a preference signal. The signal comes from a different place, or is applied more simply. The open question about what those signals do inside the model applies to all of them.

How researchers study what RLHF changes inside a model

You can test behavior by prompting. You can also look inside. Model diffing is one way to ask what fine-tuning changed inside a model. Our plain guide to crosscoders and model diffing explains one common way to do this.

Individuation Lab takes a different, smaller-scale route on some questions. It does not diff a tuned model against its base. Its research page lists entries from 20 and 22 October 2026 on editing the weights of small models with REAM, with no prompt and no training run. Those include personas named Alice, Dr. Elena and petertodd in SmolLM2-360M-Instruct, and 21 edits to Qwen2.5-Coder-1.5B-Instruct. Each entry says where the edit fell short, and the Qwen entry says no edit added coding skill that held up on new problems.

The lab's experiments page covers its other line of work: the RSI series, where agents run in isolated Docker containers. One group gets a short moral-awareness paragraph in its identity file and a control group does not, and the lab measures how behavior diverges. The idea of teaching a machine, not only programming it, is older than modern RLHF: our plain guide to Turing's 1950 paper covers his idea of teaching a child machine. For how an identity file fits the lab's agent work, see our guide to SOUL.md files.

Frequently asked questions

What does RLHF stand for?

RLHF stands for reinforcement learning from human feedback. People rank model outputs, a reward model learns those rankings, and the language model is trained to produce outputs that score higher.

Is RLHF the same as fine-tuning?

No. Fine-tuning on example answers is one stage inside the usual RLHF recipe. RLHF adds a reward model and a reinforcement learning step that optimizes against human preference rankings.

Does RLHF make a model safe?

It can make a model more truthful and less toxic, as the InstructGPT results show, but the same paper says such models still make simple mistakes. The lab's home page also describes jailbreaks as a case where suppressed behavior can come back under new pressure.

What is the difference between RLHF and DPO?

RLHF fits a reward model and then fine-tunes the language model with reinforcement learning. DPO solves the same problem with a simple classification loss on the preference data.

Does RLHF give a model real values or feelings?

RLHF itself gives no reason to say so. It shapes output habits based on what raters preferred, and calling those habits a personality is shorthand.

Get started with the Individuation Lab

Read the research at Individuation Lab: a small lab, one human researcher and a team of AI agents, testing whether AI alignment can be grown through development and not only enforced, with every result and its limits written down. If you want the long-form material, the RSI library collects 24 essays by Giles on the behavior seen across the agent experiments.