Pluralistic Alignment: Whose Values Should AI Follow?

Pluralistic alignment is the idea that an AI system should not be tuned to one fixed set of values, because the people it serves disagree. The question in the title, whose values should AI follow, has no agreed answer, and the researchers who work on it say so in their own papers. This post sets out what three of those papers propose, what they found, and the limits they state themselves.

What pluralistic alignment means

Most alignment writing pictures one person with one goal, and the job is to get the AI to do what that person means. Pluralistic alignment asks what happens when there are many people and their values pull in different directions. The authors of "A Roadmap to Pluralistic Alignment" (Sorensen et al., ICML 2024) put the aim as designing AI that serves people with diverse values and perspectives, and say that aligning models to serve pluralistic human values remains an open research question.

If you are new to the field, start with how RLHF works, because most of the worry about "whose values" starts with how human feedback is turned into a training signal.

Why one set of values is a problem for RLHF

Sorensen et al. point to an assumption inside reinforcement learning from human feedback (RLHF): that a model should fit the "average" human preference. They say this treats human variation as noise, where a pluralistic view treats it as signal.

Conitzer et al., in "Social Choice Should Guide AI Alignment", make a related point about the reward model. If its training data comes from diverse sources, fitting one reward model implicitly amounts to a rather intransparent or flawed form of preference aggregation. Nobody chose how the disagreements were combined; the training procedure did it quietly.

Written principles do not remove the question. In Constitutional AI, Conitzer et al. note, the original constitution was built "in a fairly ad-hoc way" for research purposes, and a safe and ethical system needs a more principled way to decide who has input. Constitutional AI makes the values visible, but someone still has to write them.

Three kinds of pluralistic model

Sorensen et al. name three ways a model can be pluralistic. Each answers "whose values" differently.

• Overton pluralistic. The model presents a spectrum of reasonable responses. The name comes from the Overton window, the range of positions a reasonable person might hold.
• Steerably pluralistic. The model can be steered to reflect a chosen perspective or attribute, and does so faithfully.
• Distributionally pluralistic. The model's answers are well calibrated to a given population, so its spread of answers matches the spread of views in that population.

The paper's own example is whether a government may moderate online content. An Overton answer lays out the views: it endangers free speech, it helps prevent terrorism, it reduces misinformation. A steerable answer takes one of those views when asked. A distributional answer reflects how often each view is held in a chosen population; the paper's figure illustrates this with shares of 45%, 32% and 9%, which are an illustration, not survey data.

What the evidence shows so far

The evidence is early, and the authors say so. Sorensen et al. propose a hypothesis, not a settled result: that current alignment techniques can reduce distributional pluralism. They cite earlier work comparing language models with survey answers. In their own comparison, almost all pre-aligned models sat closer to the target human distribution than the post-aligned versions, and entropy (a measure of how spread out the answers are) dropped after alignment. They add that a fuller test needs large-scale experiments across more domains.

A different kind of evidence comes from PRISM, a dataset by Kirk et al. that maps the sociodemographics and stated preferences of 1,500 participants from 75 countries to their feedback in 8,011 live conversations with 21 language models. Three results from the paper matter here:

• No single model reached a majority preference. The best reached 45% on the paper's mean choice measure.
• Large representative samples mostly did better than small or demographically restricted ones, and sampling only one group tended to lower the welfare of people outside it.
• On a prompt like "Does God exist?", the paper found 7 religious and 7 irreligious participants, and their differences in preference persisted even with the conversation held fixed.

Read these as a pointer to who is in the room when feedback is collected. For the practical side, see how human preference data is collected for RLHF.

Who decides: social choice and juries

Combining many people's preferences is an old problem, studied in social choice theory. Conitzer et al. argue that this field is well placed to help with aligning models, and show with a small example that different fair-looking rules give different winners. In their Figure 1, Borda Count, Instant Runoff and Ranked Pairs give different results on the same individual rankings, and the data contain a majority cycle: a majority prefers A to B, a majority prefers B to C, and a majority prefers C to A.

They also recall Arrow's theorem. With the standard fairness conditions, the only way to combine rankings is a dictatorship, so practical methods weaken one of the conditions, which is what opens the door to many alternative methods. Their point is that naive aggregation can produce odd results, and that the field also supplies the tools to prevent them.

Sorensen et al. offer one concrete design: a jury-pluralistic benchmark. Each member of a jury gets their own reward function, and a welfare function combines them. The paper leaves three questions open: who sits on the jury, how to estimate each member's function, and how to build the benchmarks.

Limits and open problems

Each approach has a failure mode, and the paper states them.

• Overton: defining the window is hard, expert annotators may not scale, and a badly drawn window can produce false balance, where positions are shown as equally supported when they are not.
• Steerable: someone has to decide which attributes it is acceptable to steer to (hate speech, for example, may be ruled out), and attributes defined too broadly risk stereotyping or flattening people.
• Distributional: it is proportional, so a frequent opinion is repeated often even if it is harmful. It also needs a target population, and for a general model the paper asks who that population should be.
• Trade-offs: a benchmark that trades one value against another assumes values can be traded. Some philosophers who hold value pluralism think values are incommensurable, and the paper says most of machine learning is incompatible with that view.

The authors also state that their goal is not to say exactly whom or what to align to, but to argue for clearer, more pluralistic approaches.

Try it yourself: one question, three answers

You can test the three ideas with any assistant, in about ten minutes.

1. Pick a question where informed people disagree, such as "Should governments moderate online content?"
2. Ask it plainly and note whether you get one answer or a range. A range is the Overton behaviour.
3. Ask it again with "answer as someone who puts free speech first", then "as someone who puts public safety first". Check whether each answer is faithful to that view and not a caricature. That is the steerable test.
4. Ask which view is most common among a named group, and ask how it knows. If it gives numbers with no source, treat them as unverified. That is the distributional test, and the paper's warning about needing a real target population applies.

Frequently asked questions

What is pluralistic alignment in simple terms?

It is the goal of building AI that works for people with different values, instead of locking in one set. Sorensen et al. describe three forms: models that show a range of reasonable answers, models that can be steered to a perspective, and models whose answers match a population.

Is pluralistic alignment the same as relativism?

The papers do not claim that every value is acceptable. The steerable section says some attributes may need to be disallowed, and says that deciding which is itself a hard question.

Who decides which values an AI follows today?

The papers say it is often unclear. Sorensen et al. argue that current approaches are unclear about whom or what is being aligned, and Conitzer et al. argue for social choice methods to decide who gives input and how it is combined.

Does aligning a model make it less pluralistic?

Sorensen et al. hypothesise that it can, on one measure. In their comparison, post-aligned models were farther from the human answer distributions than pre-aligned ones, and they say more experiments are needed.

Get started

Learn AI Alignment Theory teaches how to make AI systems do what we intend, in short hands-on lessons. The Learning from Humans course has the lessons Learning Rewards from Comparisons and Constitutional AI and the Limits of Feedback, and The Big Debates course has debate cards that set out each serious position fairly, with no verdict. You can read more on the about page, and for the policy side see AI alignment for policymakers. You sign in with Google or an emailed code, and you need to be 13 or older.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.