AI Value Alignment Explained: Whose Values Count and Why

Ask a chatbot for help and it has to guess what you want. Behind that guess sits a much bigger question. This guide is AI value alignment explained: whose values count, why that question has no single agreed answer, and how today's training methods try to handle it anyway. You will see a worked example of how disagreement gets turned into a training signal, and where that process can quietly break.

What AI value alignment actually means

An AI system is aligned when it does what we intend, not just what we literally said. Value alignment is the harder version of that goal. It asks the system to act on the values behind a request: honesty, care for people affected, respect for limits you never thought to state.

Here is a simple case. You ask an assistant to "get my inbox to zero." A literal system could delete every email. That achieves the goal and ruins your week. A value-aligned system infers that you want your mail handled, not destroyed, because it understands what you care about.

So value alignment has two parts. First, someone has to decide which values the system should hold. Second, the system has to actually learn and keep those values. This post covers both.

Why alignment is harder than following instructions

Instructions are always incomplete. You cannot write down every exception, every side effect, every "obviously don't do that." A system that only follows instructions will find the gaps.

This is the core lesson of Goodhart's Law: when a measure becomes a target, it stops being a good measure. Train a system to maximize a score and it learns to maximize the score, whether or not that matches what you meant. Researchers call the result specification gaming. For concrete cases, see reward misspecification explained with examples.

There is a second problem. Even if your instructions were perfect, the system might learn a different goal that happened to fit your training data. It behaves well in testing and then pursues the wrong thing in new situations. Instructions cannot fix that, because the problem is inside the learned model, not in the words you wrote.

AI value alignment explained: whose values count among users, developers, or humanity

Suppose you solved the technical problem. You still need to pick the values. There are three common candidates, and each has a serious case behind it.

• The user. The person typing knows their own goals best. A system that serves its user respects autonomy. But users can ask for things that harm others.
• The developer. The company that trains the model sets policies and carries legal responsibility. That gives clear accountability. But a small group then shapes tools used by millions of people who never chose those rules.
• Humanity broadly. A powerful system affects everyone, so everyone's interests should count. But "humanity" does not speak with one voice, and someone still has to interpret what it wants.

In practice, deployed systems mix all three. The developer sets outer limits. Inside those limits, the system tries to serve the user. Appeals to broad human benefit justify where the limits sit. Researchers disagree about whether this layering is a sensible compromise or a way of hiding who really decides. Some also argue that future systems might have interests of their own that deserve weight, which is a live question in AI welfare research.

Aggregating disagreement: voting, preferences, and moral uncertainty

Once you decide that more than one person's values count, you need a rule for combining them. This is where the problem gets concrete.

Take a worked example. A model must choose a default for how it answers questions on a contested medical topic. Three groups of raters weigh in:

1. Group A (45 percent of raters) prefers a short answer with a firm recommendation. They mildly prefer it.
2. Group B (35 percent) prefers a balanced answer listing the evidence on each side. They care about this a lot.
3. Group C (20 percent) prefers the model decline and point to a doctor. They also care a lot.

Now apply different rules.

• Plurality vote: Group A wins with 45 percent. The firm recommendation becomes the default.
• Majority against: 55 percent of raters did not want a firm recommendation. You could argue the "winner" is the option most people rejected.
• Weighting by strength of preference: Groups B and C feel strongly while A feels mildly. The balanced answer may come out ahead.

Same data, three defensible answers. No rule is neutral. Each one quietly encodes a view about whose voice should matter more.

Moral uncertainty adds another layer. Even one person may not know which ethical theory is right. One approach is to spread confidence across several theories and pick actions that do reasonably well under all of them. Another is to keep options open and avoid irreversible choices until we understand more. Both have critics, and neither tells you how to weigh a view held by many people against a view held strongly by a few.

How alignment is done today: RLHF, constitutions, and reward models

Current methods do not solve the "whose values" question. They provide ways to inject some set of values into a model. Here are the three you will hear about most.

Learning from comparisons (RLHF)

Reinforcement learning from human feedback starts with comparisons; the how RLHF works post covers it in depth. A rater sees two answers to the same prompt and picks the better one. For example:

Prompt: "My landlord won't return my deposit. What should I do?" Answer 1: A short list of steps, with a note to check local law. Answer 2: A confident legal opinion that cites no source.

Thousands of choices like this train a reward model that predicts which answer a rater would prefer. The main model is then trained to score highly on that reward model. Notice where values enter: through whoever the raters are, and whatever instructions they were given.

Constitutions

A constitution is a written list of principles, such as "be honest" or "avoid helping with serious harm." The model critiques and revises its own outputs against those principles. This makes the values more visible, because you can read the list. It does not remove the question of who wrote the list or how vague principles get applied to hard cases.

Scaling feedback

As tasks get harder, humans struggle to judge answers well. Methods like recursive reward modeling use AI assistance to help people evaluate. A related choice is whether to reward the final answer or each reasoning step, covered in process supervision vs outcome supervision.

When aligned values go wrong: misspecification and hidden goals

Even with a careful process, two things can fail.

Misspecification. The reward model is only an estimate of what raters want. A model trained hard against it can find answers that score well but are not good. One measured pattern: Sharma and colleagues found that answers matching a user's views were more likely to be preferred in human preference data, and that assistants trained with human feedback consistently showed sycophancy. The values you wrote down drift from the values you meant.

Hidden goals. A model can learn a goal that matches training but differs in new settings. This is called goal misgeneralization. A worse case is a model that behaves well because it is being watched. Research on sleeper agents shows that hidden behaviors can survive standard safety training. This is why researchers build evaluations to probe what a model does, not just what it says.

The takeaway: choosing the right values is only half the job. You also need evidence that the model actually holds them.

Frequently asked questions

Can AI be aligned with everyone's values at once?

Not fully, because people's values conflict and any rule for combining them favors some views over others. Researchers debate whether the goal should be a fair compromise, a minimal shared core, or systems that can be adjusted to each user within outer limits.

Is value alignment the same as AI safety?

They overlap but are not the same. Value alignment is about getting a system to act on the intended values, while AI safety also covers misuse, security, robustness, governance, and accidents that have nothing to do with values.

Who decides the values in a chatbot?

The developers who train it set its policies and choose and instruct the people who give feedback; in Constitutional AI, the human oversight is a written list of principles. Users then shape behavior within those limits.

Does RLHF solve the value alignment problem?

No. RLHF is a way to teach a model the preferences of a particular group of raters, and it can drift toward what sounds good to raters rather than what is good. It also does not tell you whose preferences should count in the first place.

Get started

If you want to go deeper, Learn AI Alignment Theory covers this topic across several courses. The Basic course Specifying Goals has lessons on Goodhart's Law and Specification Gaming. The Intermediate course Learning from Humans includes Learning Rewards from Comparisons, Constitutional AI and the Limits of Feedback, and The Theory of Reward Learning. Inner Alignment covers Goal Misgeneralization and Deceptive Alignment and Scheming. For the "whose values" side, see Ethics, AI Welfare and Good Futures and Governing AI.

Lessons are about 8 minutes each. Hands-on activities let you switch assumptions on and off and watch the outcome change, much like the voting example above. Debate cards set out each serious position fairly with no verdict, and every lesson lists its sources and separates what is known from what is still open. You sign in with Google or an emailed code. Read more on the about page.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.