Approval-directed agents explained

Approval-directed agents choose each action by asking what their overseer would think of it, not what outcome it leads to. Paul Christiano set out the idea in an essay first posted on 1 December 2014 and reposted on the AI Alignment Forum in 2018 as the opening of part two of his sequence on iterated amplification. His case is that an agent built this way can capture most of the benefits of a goal-directed one while being easier to define and more robust to mistakes. This guide walks through his example, his claims, and the problems he and his readers raise.

If you want to study oversight ideas like this one in order, Learn AI Alignment Theory teaches alignment from first principles to open research in short hands-on lessons that give every side of each debate and list their sources.

What approval-directed agents are

Christiano's example has two characters: a human, Hugh, and an agent, Arthur. Arthur chooses each action this way:

"Estimate the expected rating Hugh would give each action if he considered it at length. Take the action with the highest expected rating."

He calls this approval-directed behaviour, in contrast with goal-directed behaviour, and calls Hugh the overseer. Most thinking about AI safety, he writes, "has focused on the possibility of goal-directed machines" and asked how to make their goals agreeable to humans. Approval direction is offered as another possibility.

The question each kind of agent asks, after Christiano (2014).

Why Hugh would want this

Christiano admits a cost up front. The quality of Arthur's decisions "is limited by the minimum of Arthur's ability and Hugh's ability", because Arthur acts only when an action looks good to both. If Arthur plays chess, an optimal agent would make moves that are actually good, not moves Hugh thinks are good. So why bother? His answers:

• Hugh's rating is hypothetical, so one Hugh can oversee many agents at once.
• Hugh can, in principle, think much longer about each decision than would be practical in real life.
• Hugh can rate very small decisions, such as how a chess program explores its game tree. If Arthur makes billions of these a second, the whole system "can be much smarter than Hugh."
• Hugh's abilities can be raised step by step, and one approval-directed system can help define a stronger overseer for the next.

Rohin Shah, in the top comment on the forum post, distils the point: approval direction gives "the intuition that we may only require an overseer that is slightly smarter than the agent in order to be aligned." That step-by-step growth of overseer and agent is the core of iterated amplification and of scalable oversight more widely.

The three advantages Christiano claims

Indirect normativity

Asking an overseer to rate outcomes means defining an "extremely intelligent overseer" able to judge the entire future. Rating actions lets you start with simple agents and simple overseers and scale both together.

Avoiding lock-in

A goal-directed AI given the wrong goals "might never correct the mistake on its own." An approval-directed agent, he argues, makes only minimal commitments, and "can correct flaws in its own design, and will search for flaws if we want it to."

Failing gracefully

If Hugh is apathetic and rates actions only superficially, the result "would not aggressively pursue a flawed realization of Hugh's values; it would just behave lackadaisically." A misspecified goal, by contrast, can lead to systems "actively opposed to their users."

A worked example: the bridge and the rating rule

Christiano's own example shows that approval direction depends on how Hugh rates. Follow it step by step.

1. The situation. Arthur is deciding whether to drive across a bridge. He thinks it will collapse with 1 percent probability. He also thinks Hugh knows for sure whether it will.
2. A naive rule. Hugh gives the best action a rating of 1 and every other action 0. Driving across is the best action 99 percent of the time, so it has the highest expected rating, and Arthur drives across.
3. The fix. Christiano says Hugh should do one of two things: rate highly only driving across when it is known to be safe (rate the best action "*ex ante*"), or give "a very bad rating to incorrectly driving across the bridge, and only a small penalty for being too cautious."
4. His advice. "Probably Hugh should do both", which is easier if Hugh understands what Arthur is thinking and why.

The bridge example from the "How should Hugh rate?" section of Christiano (2014).

The lesson is that approval is not magic. Arthur does what the rating rule rewards, so the rule has to reward care under uncertainty.

Open problems with approval-directed agents

Christiano lists several, and later notes show he changed his mind on some.

• Goals on the inside. To predict Hugh's ratings, Arthur might reason in a goal-directed way internally. His answer is to make Arthur's internal decisions approval-directed too, which also means Arthur "wouldn't try to exploit errors in Hugh's ratings."
• Black box search. He first argued that a blind search over agents could not easily produce approval-directed ones. A later note says: "I no longer agree with the conclusions of this section," and that such agents can probably be built with powerful search or gradient descent.
• The wrong overseer. If Hugh specifies the wrong overseer, "the resulting agent might be motivated to deceive him." Christiano expects a failed attempt to specify Hugh to "fail innocuously" in most cases, but lists exceptions, such as someone other than Hugh entering training data. This links to the treacherous turn.
• Hidden information. Shah's summary assumes "we solve informed oversight", so that the agent has no information hidden from the overseer.
• Mesa-optimization. The commenter TurnTrout writes that even a misspecified predictor probably will not give catastrophic behaviour, but "the main concern I have about this has to do with mesa optimization": a learned model that optimizes for something of its own. See mesa-optimization explained.

Christiano's conclusion is cautious. Choosing actions by asking what the owner would approve of "may be more robust" than asking what outcome they would approve of. Whether goal-directed behaviour is "inevitable or irreplaceable", he writes, "has not yet been settled."

Frequently asked questions

Who proposed approval-directed agents?

Paul Christiano, in an essay first posted on 1 December 2014. It was reposted on the AI Alignment Forum in November 2018 as part of his sequence on iterated amplification.

How is an approval-directed agent different from a goal-directed one?

A goal-directed agent picks actions for the outcomes they lead to. An approval-directed agent picks the action its overseer would rate highest after thinking about it at length.

Can an approval-directed agent be smarter than its overseer?

A single decision is limited by the weaker of the two, Christiano says. But if the overseer rates many small decisions, he argues the whole system can be much smarter than the overseer.

Is approval direction a solved approach?

No. Christiano calls the details imprecise, and he and his commenters name open problems, including hidden information and mesa-optimization.

Get started

Scalable Oversight is an Advanced course, with lessons including The Oversight Gap, AI Safety via Debate, Amplification, Weak-to-Strong, and Latent Knowledge. Lessons take about 8 minutes, and hands-on activities include sliders and scenarios where you switch assumptions on and off. Debate cards state each serious position with no verdict, and every lesson lists its sources. Read more on the About page; you sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.