Decision Theory for AI Alignment: Why It Matters

Decision theory for AI alignment starts from a plain worry. Two systems can want the same thing and still act differently, because they use different rules to score their options. This post sets out three rival rules, works one puzzle with numbers from the research papers, and shows why a rule that looks fine can turn out unstable once a system can change itself.

Decision theory for AI alignment: what the field asks

A decision theory says how an agent should turn what it wants and what it believes into a choice. For a person this is a philosophy topic. For an AI system that acts in the world, it is a design choice, because whatever rule it uses decides what it does when you did not think to test the case.

Soares and Fallenstein, in "Toward Idealized Decision Theory" (arXiv 1507.01986), argue that decision theory is necessary for aligning smarter-than-human systems. They discuss the shortcomings of two standard formulations and say neither can describe an idealized decision procedure for artificial systems to approximate. They then point to two ideas, policy selection and logical counterfactuals, as paths for future research. Those are open research claims, not settled results.

This is a different problem from the one in Goodhart's law in AI. That one asks whether the goal is right. Decision theory asks how the agent works out what each action would lead to, even when the goal is fixed.

Three ways to score an action

All three theories begin the same way: some outcomes are better than others, and the agent has beliefs about which actions lead to which outcomes. They differ in the question they ask of an action. Yudkowsky and Soares put the three questions side by side in "Functional Decision Theory" (arXiv 1710.05060):

• Causal (CDT): which physical act of mine causes the best outcome?
• Evidential (EDT): which physical act of mine would be the best news to hear I took?
• Functional (FDT): which output of this very decision procedure gives the best outcome?

In simple settings the three agree. The paper says their prescriptions match when events correlate with the agent's action only if the agent causes them. They split when something outside the agent, such as a predictor or a copy, tracks how the agent decides.

A worked example: the Newcomb sums

Here is the paper's version of Newcomb's problem. You face a transparent box A holding $1,000 and an opaque box B holding either $1,000,000 or $0. A predictor, correct 99% of the time in the past, put $1,000,000 in box B only if she predicted you would leave box A behind. She has already left. Do you take both boxes or only box B?

Work the sums yourself, using the 99% as the chance the prediction matches what you do. These two totals are my arithmetic from the paper's figures, not numbers the paper reports.

1. Take only box B. With probability 0.99 the predictor foresaw it and box B is full: 0.99 times $1,000,000 is $990,000.
2. Take both boxes. With probability 0.99 she foresaw that and box B is empty, so you get $1,000. With probability 0.01 she was wrong and you get $1,001,000. That is $990 plus $10,010, which is $11,000.

The paper says EDT reasons that leaving the $1,000 is good news, so it one-boxes and walks away richer. CDT reasons that the boxes are already filled, "my decisions now can't change the past", so it takes both and typically comes away with $1,000. The paper then flips the case with Skyrms's smoking lesion, where a lesion causes both smoking and cancer. There CDT smokes, which does better, and EDT refuses to, which does worse. The paper's claim is that FDT gets both right: one box, and smoke. Whether that settles anything is the subject of a later section.

Why a safety researcher cares which rule wins

You could call all this a parlor game. The papers give a reason it is not. Soares and Fallenstein ask what a self-modifying agent does when it can rewrite its own decision rule. Their example: a CDT agent is about to be copied and will then face its copy in a one-shot Prisoner's Dilemma. If it stays a CDT agent after the copy, it defects and gets payoff 1. If it first modifies itself to cooperate with its copy, it gets payoff 2. So CDT, asked what to do, says to stop being CDT.

The paper calls this instability under reflection. A rule that recommends replacing itself is not a rule you can write down and trust to stay. A second case in the same paper has someone steal the agent's source code and use it to predict the agent, so a Newcomb-like situation arrives from the agent's past, which causal reasoning treats as cut off from the agent.

That is why decision theory sits next to the topics on embedded agency and agent foundations. The abstract of "Embedded Agency" (arXiv 1902.09469) says traditional models treat the agent as cleanly separated from its environment, while a real agent must reason about itself as one more physical system whose parts can be modified. It names three obstacles to getting good reasoning for such agents.

Cases where other rules do better, and the critique

The FDT paper tries to show its rule wins on more than Newcomb. In the twin prisoner's dilemma, each of two agents who reason the same way gets $1,000,000 if both cooperate, $1,000 if both defect, and a defector against a cooperator gets $1,001,000. CDT defects and both end with $1,000; EDT and FDT cooperate. In Parfit's hitchhiker, a driver rescues you from the desert only if you will pay $1,000 once in the city, and she detects lies 99% of the time. The paper says CDT and EDT agents cannot honestly promise to pay, so they are left in the desert, while FDT pays.

Not everyone accepts this. The philosopher Wolfgang Schwarz refereed the FDT paper for a philosophy journal and published his report in a post, "On Functional Decision Theory". He recommended resubmission with major revisions and the journal rejected the paper. His objections, in his words and mine:

• Some of its recommendations strike him as insane, such as refusing to pay a $1 blackmailer because a person who refuses would likely not have been blackmailed.
• The paper says to suppose a mathematical function gave a different output, and he says it does not explain how to compute outcomes under that supposition.
• Whether FDT agents fare better than CDT agents depends, he argues, on the environment, on how faring better is measured and on what the agents care about. He suggests agents can also be built to cooperate by caring about others, with no new decision theory.

Both sides appear in the record, and I will not pick one. The learning app's About page says its 71 debate cards set out each serious position fairly with no verdict, and this post tries to do the same.

Open problems that remain

• Logical counterfactuals. Soares and Fallenstein say no formalization of them existed when they wrote, and Schwarz makes the same point about FDT.
• Logical uncertainty. A real agent is unsure of the results of its own long computations. "Logical Induction" (arXiv 1609.03543) presents a computable algorithm that assigns probabilities to logical statements and refines them over time. It is one proposal, not an agreed answer.
• Stability. Which rule, if any, an agent would keep after rewriting itself.
• Connection to goals. The same themes of incentives show up in the posts on instrumental convergence and the off-switch game, where an agent's rule for weighing an action decides whether it lets itself be corrected.

How to study decision theory for alignment

1. Redo the Newcomb sums above with a different accuracy, such as 60% and 51%. Find where the two totals cross, and note what you assumed about the predictor.
2. Write the three questions from the diagram on one page. Run the twin prisoner's dilemma through each and check your answers against the paper.
3. Read the FDT paper's Newcomb and hitchhiker sections, then Schwarz's post, and write the strongest version of the side you disagree with.
4. Read how the same rule questions appear in the posts on coherence theorems and corrigibility.

In Learn AI Alignment Theory, Agents and Incentives is a Basic course, and Agent Foundations and Cooperative AI and Multi-Agent Safety are Advanced courses. Those names are all this post claims about them.

Frequently asked questions

What is decision theory in AI alignment?

It is the study of how an agent should turn its goals and beliefs into a choice, applied to artificial systems. Soares and Fallenstein argue it is necessary for aligning smarter-than-human systems.

What is the difference between causal and functional decision theory?

Causal theory scores the physical act by what it causes. Functional theory asks which output of the agent's decision procedure gives the best outcome, which matters when a predictor or copy tracks that procedure.

Is there one correct decision theory for AI?

No, and researchers disagree. Schwarz's published critique of functional decision theory shows the dispute is live.

Why does self-modification matter here?

Because a rule that tells the agent to replace itself, as the paper shows for causal theory, cannot be assumed to stay in place.

Get started

Learn AI Alignment Theory has 22 courses and 69 lessons of about 8 minutes, each listing its sources, across three levels from core ideas to open research. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.