Coherent extrapolated volition (CEV) explained

Coherent extrapolated volition, or CEV, is an idea about what a superintelligent AI should aim for. Its answer: not what people want today, but what they would want if they knew more, thought faster and had grown up farther together. Eliezer Yudkowsky proposed it in 2004 as part of his work on friendly AI. This post explains what CEV says, why its backers like it, and the open problems critics raise, with no verdict. The claims come from Wikipedia's Coherent extrapolated volition and Friendly artificial intelligence articles.

What coherent extrapolated volition means

Wikipedia calls CEV a theoretical framework in the field of AI alignment. It is one answer to a hard question in that field: whose goals should a very capable AI pursue, and which version of them?

Under CEV, an artificial superintelligence (ASI) would act on a kind, careful guess about what humans could want if they were:

• more knowledgeable,
• more rational,
• given more time to think, and
• matured together as a society.

That is the opposite of following people's current wishes, one person at a time or as a crowd. If you are new to the idea of an ASI, start with our post on artificial superintelligence.

The three words, one at a time

The name packs the whole idea into three words. Read them right to left.

• Volition is what people will, or wish for.
• Extrapolated means projected forward: not the wish as it is now, but as it would be under better conditions of knowledge and reflection.
• Coherent means the AI acts where those projected wishes agree, not where they clash.

Yudkowsky put it in a single, often quoted sentence. Our coherent extrapolated volition is "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted".

How CEV would work, in principle

The Friendly artificial intelligence article describes the plan. A friendly AI would not be designed directly by human programmers. Instead, a "seed AI" would be programmed to first study human nature. It would then produce the AI that humanity would want, given enough time and insight.

In the CEV article's words, the AI would aggregate and project human preferences into what people could desire under ideal conditions of knowledge and morality. The aim is an AI that:

• does not trespass on humanity's true interests,
• does not follow transient or poorly informed preferences, and
• does not decide questions on which humanity's will is still unclear.

There is a catch the Friendly AI article points out. Extrapolated volition is meant to be what humanity would want, all things considered. Yet it can only be defined starting from people as they are today, with today's minds.

Why its backers find CEV appealing

Yudkowsky and Nick Bostrom list several properties they find attractive:

• It is self-correcting. It captures the source of human values instead of trying to list them.
• No fixed rulebook. It avoids laying down an explicit, fixed list of rules.
• Room to grow. It allows for moral growth, so flawed beliefs of today do not get locked in. Our post on value lock-in explains why that matters.
• Less power to the builders. It limits how much a small group of programmers can shape what the ASI values, which also reduces the reward for building ASI first.
• People stay in charge. It keeps humanity in charge of its own destiny.

The open problems

The CEV article also says the idea faces significant theoretical and practical challenges.

Whose volition counts?

Bostrom notes that CEV has "a number of free parameters that could be specified in various ways, yielding different versions of the proposal." One is the extrapolation base: whose wishes are counted at all. Should it include people with severe dementia, patients in a vegetative state, foetuses or embryos? And if only humans count, the result might be ungenerous toward other animals and digital minds. One suggested fix is a way to widen the base over time. For the wider debate on whose values an AI should follow, see our post on AI value alignment.

Can ideal values be pinned down?

The Friendly AI article reports a related criticism. Writing in AI & Society, Boyles and Joaquin question proposals for machines that reason about the moral values people would have had. They point to the endless number of "what if" conditions that would need programming, and the difficulty of spelling out values more ideal than the ones people hold now.

Is any of this needed?

Others doubt the starting point. Some critics think human-level AI and superintelligence are unlikely. Alan Winfield, writing in The Guardian, says we need to be "cautious and prepared" but "don't need to be obsessing" about superintelligence. And some philosophers claim any truly rational agent will be benevolent anyway, so careful safeguards may be unneeded or even harmful.

Variants and alternatives

CEV is not the only proposal. One alternative is to let an ASI use its greater intelligence to work out what is morally right, and act on that. The two can be mixed: an ASI could follow CEV except where doing so would be morally impermissible.

Another variant comes from a philosophical analysis that draws on Anthony Giddens' idea of "active trust". It proposes "Coherent, Extrapolated and Clustered Volition" (CECV), meant to better reflect the moral preferences of different cultural groups.

A worked example: run one wish through the four questions

CEV is meant for a whole civilization, but you can feel how it works on a small scale. This is a thought exercise, not how an AI would compute anything.

1. Pick one wish you hold today. For example: "My town should ban cars from the main street."
2. If you knew more: what would you learn about deliveries, shops or people who cannot walk far? Write down whether the wish changes.
3. If you thought faster and longer: what side effects would you notice after a week of reflection?
4. If you were more the person you wish you were: would a calmer or fairer version of you want the same thing?
5. If you had grown up farther together with your neighbours: where would your wish and theirs end up agreeing?

Now look at what is left. Maybe "a quieter, safer main street" survives every question while "ban all cars" does not. That surviving core is a tiny picture of what "coherent" means. Notice, too, the hard parts: you had to guess how others would change, and decide whose wishes to include. Those are exactly the open problems above.

Exploring the idea in Contain ASI

Contain ASI is a story strategy game for 1 to 4 players that runs in your browser and needs WebGL 2. Chapter 1, Before ASI, starts in January 2024 with four fictional AI labs in one race. As a researcher, research manager, CEO or government, you try for three years to keep every lab from crossing into recursive self-improvement before 2027.

The game is fiction, not a forecast: the page says all its labs, people and events are fictional. Ideas like CEV are about what a superintelligence should want once it exists. The game asks an earlier question: can anyone slow the race long enough for such questions to be answered? For the history of the term behind CEV, see our post on friendly AI.

Frequently asked questions

What does coherent extrapolated volition mean in simple words?

It means an advanced AI should act on what people would want if they knew more, thought faster and had grown up farther together, and only where those wishes agree. It is not what people happen to want today.

Who came up with CEV?

Eliezer Yudkowsky proposed it in 2004 as part of his work on friendly AI. Nick Bostrom has also noted its properties and its open problems.

What is the extrapolation base?

It is the group whose wishes CEV takes into account. Bostrom notes the choice is open: it could include or leave out certain people, other animals and digital minds.

Is CEV a working system?

Wikipedia describes it as a theoretical framework, and reports that it faces significant theoretical and practical challenges.

Get started

Curious how hard it is to buy time for questions like these? Open Contain ASI in your browser and start a campaign as a researcher, research manager, CEO or government.