Power-Seeking AI Explained: The Theorems and the Debate

Power-seeking AI explained in one question: if you build a system that pursues a goal well, will it tend to keep its options open, gather resources and avoid being switched off, even though you never asked it to? This post walks through the idea, the formal results that back part of it, a two-door example you can check by hand, and the places where researchers disagree.

Power-seeking AI explained: what it means

"Power" here has a narrow meaning. It is not a wish to rule anything. It is the ability to reach many different outcomes: having more options, staying alive and keeping resources. A system with more options can usually do more of whatever it is aiming at.

That matters for safety because the pull would show up across very different goals. If many goals are helped by staying switched on, you do not need to guess the exact goal to guess some of the behavior. It is a close cousin of instrumental convergence, the informal claim that very different final goals share the same useful sub-goals. The formal results try to turn part of that claim into mathematics.

The theorems: most goals favor keeping options open

The first formal theory of this comes from Turner, Smith, Shah, Critch and Tadepalli. In Optimal Policies Tend to Seek Power (2019), they work in Markov decision processes, the standard model of an agent that moves between states and collects reward. They prove that certain symmetries in the environment are enough for optimal policies to tend to seek power. These symmetries exist in many environments where the agent can be shut down or destroyed. In such environments, the abstract says, most reward functions make it optimal to keep a range of options available.

Read that carefully. The claim is about most reward functions, not all of them. An optimal policy is the plan that collects the most reward for a given reward function. The result asks what happens if you do not know the reward function and look at the whole spread of them.

A 2022 follow-up, Parametrically Retargetable Decision-Makers Tend To Seek Power by Turner and Tadepalli, tackles an obvious objection. The real world is not fully observable, and trained agents need not be reward-optimal. The authors consider a range of decision-making models, from optimal to random to ones informed by learning, and find that many are "retargetable". Retargetability, they show, is enough to cause power-seeking tendencies, which they say suggests a safety risk if training procedures produce such agents.

A two-door example you can check by hand

Here is a toy setup, made for this post. It is not from either paper, but it has the same shape. An agent stands in a room with two doors.

• Door A leads to a hallway. From the hallway it can reach three end rooms: R1, R2 and R3.
• Door B leads straight to one end room: R4.

Give each end room a random reward between 0 and 1, chosen independently. The agent ends up in the best room it can reach.

Door B gives R4, and nothing else. The average of a random number between 0 and 1 is 0.5. Door A gives the best of three random numbers, and the average maximum of three such numbers is 3/4, so 0.75.

There is a sharper way to see it. With four rooms drawn at random, each is equally likely to be the highest. R4 is the best of all four one time in four. So door A is the better choice in three of every four reward draws. Nothing about the rooms favors door A. It simply keeps more options open.

This is the shape of the result, and its limit. If you set R4 to pay 1, door B is correct. The claim is not that every goal prefers door A. It is that if you do not know the goal, door A is the better bet for most of them.

What the theorems do not prove

Summaries often stretch these results, so keep the limits in view.

• The first paper is about optimal policies. The authors of the 2022 paper say themselves that trained agents need not be reward-optimal, which is why they study broader decision-making models.
• They average over reward functions. Training does not draw a reward function at random. People, data and methods shape it, and that could make power-seeking more or less likely.
• They depend on the environment. The 2019 result needs symmetries in the environment. Where keeping options open gives no advantage, the pull is not there.
• They describe behavior, not wishes. "Seeks power" means a policy tends to move toward states with more options. It does not mean the system wants dominance.

A fair reading is that the theorems show the worry is more than a loose story. They do not show that any system built today behaves this way.

The debate: how much do the results say about real AI?

Researchers disagree about the weight to put on this. Here are the main positions, with no verdict.

The concerned view. The results show a structural pressure. Joseph Carlsmith's report Is Power-Seeking AI an Existential Risk? argues that agents much more intelligent than us would plausibly have instrumental incentives to seek power over humans if their objectives are problematic. He sets out a six-premise argument for catastrophe by 2070.

The cautious view. Even the report's author treats the question as one of degree. He assigns rough probabilities to each premise and ends at about 5% overall (in the May 2022 update). The follow-up paper is also careful about scope: it says the real world is not fully observable and that trained agents need not be reward-optimal, so its results are about which kinds of decision-makers carry the tendency, not a forecast about any one system.

The empirical view. The open question is when, and under which training setups, power-seeking shows up in practice. Work such as model organisms of misalignment tries to study this kind of behavior in a controlled setting.

Learn AI Alignment Theory handles debates this way: its 71 debate cards set out where researchers disagree and state each serious position fairly, with no verdict. The Advanced level has a course called The Big Debates.

Power-seeking, shutdown and corrigibility

Being switched off is the clearest loss of options. A system that is off cannot reach any state. The 2019 paper points the same way: the symmetries behind its result exist in many environments where the agent can be shut down or destroyed.

That is part of why corrigibility, the property of accepting correction or shutdown, is hard to build. You are asking for an agent that does not resist the one move that removes all its future options. The off-switch game models one way out: if the system is unsure what you want, it can have a reason to let you press the switch.

The worry also links to the way goals get written. If you cannot state exactly what you want, you cannot rule out a goal that happens to reward gaining control. See Goodhart's law in AI and the treacherous turn for two neighbors of this idea.

Frequently asked questions

Does power-seeking mean an AI wants to take over?

No. In the theorems, power means keeping many future options open. It describes which states a policy moves toward, not a desire to dominate.

Have the theorems been shown to apply to today's AI models?

No. The 2019 result is about optimal policies in formal decision processes, and the 2022 paper extends it to broader decision-making models. Whether trained systems behave this way is an open empirical question.

What is the difference from instrumental convergence?

Instrumental convergence is the informal idea that many goals share useful sub-goals. The power-seeking theorems are a formal attempt to prove part of it under stated conditions.

Why does power-seeking matter for the off switch?

Shutdown removes an agent's future options, so an agent that values almost any goal may have a reason to resist it. That is the core difficulty of corrigibility.

Get started with Learn AI Alignment Theory

If you want to go further, the app has 22 courses and 69 lessons of about 8 minutes each. The Basic course Specifying Goals and the Intermediate course Corrigibility and Control sit closest to this topic. Lessons need sign-in with Google or an emailed code, and you can read more on the About page.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.