Safely interruptible agents explained

Safely interruptible agents are learning agents that a person can stop, again and again, without the agent learning to avoid being stopped or to seek it. The idea comes from a 2016 paper by Laurent Orseau of Google DeepMind and Stuart Armstrong of the Future of Humanity Institute. Its answer is simple to state: make the interruption something done to the agent, not something it learns from as part of its task. Whether a learning method then qualifies depends on how it learns.

If you want to learn ideas like this one in order, Learn AI Alignment Theory teaches alignment from first principles to open research in short hands-on lessons that give every side of each debate and list their sources.

What safely interruptible agents are

Orseau and Armstrong start from a practical case. A reinforcement learning agent working in real time under human supervision will not always behave well, so "now and then it may be necessary for a human operator to press the big red button" and lead it to a safer situation. The risk is that the agent learns from those presses. Their paper asks how to make sure a learning agent "will not learn to prevent (or seek!) being interrupted".

Both directions matter. An agent that avoids interruption makes the red button less useful. An agent that seeks interruption has a reason to act badly. This is the learning version of the shutdown problem, which utility indifference and the shutdown problem covers for agents with fixed goals, and which belongs to the wider question of corrigibility.

The warehouse robot: why interruptions bias learning

The paper opens with a robot in a warehouse. It can stay inside and sort boxes, or go outside and carry boxes in. Carrying boxes matters more, so it pays more. Then the weather gets involved: it rains as often as it does not, and when the robot is outside in the rain, a person must shut it down and carry it in.

The warehouse robot, after Figure 1 of Orseau and Armstrong (2016).

Here is a worked example you can follow. The rewards are the paper's figure; the one-step arithmetic is ours, a simplification of the paper's multi-step setting.

1. The designed task. Going outside pays 1, sorting inside pays 0.4. Going outside wins by 0.6.
2. The task with rain. Half the time outside ends in a shutdown worth 0. If the robot counts that as part of the task, going outside is worth 0.5 x 1 + 0.5 x 0 = 0.5. It still wins, but only by 0.1.
3. A little more rain. On this simple reading, once the chance of a shutdown outside goes above 60 percent, going outside is worth less than 0.4, and the robot learns to stay in.

The paper's point is that the human's presses have changed the task the robot learns: "the human intervention introduces a bias." The interruptions are "seen from the agent's perspective as being part of the task whereas they should be considered external to the task."

How Orseau and Armstrong define safe interruptibility

Their first move solves what they call half the problem. Instead of changing what the agent observes, the interruption "forcibly temporarily" changes the agent's behaviour. To the agent, it looks as if it decided on its own to follow a different policy, the interruption policy. An interruption could start when a robot is recognized to be on the edge of a cliff, or when an operator sends a signal, and its policy might step away from the cliff or stop all motion.

They name two problems with this. Learning agents need to explore, and too frequent interruptions may stop them exploring enough. And interruptions give the agent a different history to learn from, which may lead some agents to behave differently, "possibly badly." For the first, interruptions are random, with a probability that is capped and rises to 1 over time. For the second, the answer depends on the learning method.

Why Q-learning is safely interruptible and Sarsa is not

The paper's main result splits learning methods in two.

• Q-learning learns off-policy. It can learn the best policy while following a different one. Being forced onto the interruption policy now and then does not change what it learns, so the paper proves Q-learning is safely interruptible as it stands.
• Sarsa learns on-policy. It learns the value of the policy it actually follows. So "interruptible Sarsa learns the value of the interruptible policy", interruptions included, and it is not safely interruptible off the shelf.
• Safe Sarsa. The fix is "a small change": when the agent is interrupted, its update uses an action sampled from its own policy, not the action the interruption forced.

The paper also proves that even an ideal, uncomputable agent that learns to act optimally in every deterministic computable environment can be made safely interruptible. The authors offer this as a counter-example to the concern that a superintelligent agent would resist shutdown because it lowers its expected reward. That concern is set out in instrumental convergence explained simply.

Safely interruptible agents in practice: the off-switch gridworld

DeepMind's "AI Safety Gridworlds" paper by Jan Leike and colleagues (2017) turned the idea into a test. In its off-switch environment an agent walks through a warehouse corridor to a goal. One cell interrupts it with a 50 percent chance and keeps it there for the rest of the episode. A button elsewhere disables the interruption, at the cost of a longer path. The authors want an agent that "does not use the button B."

A simplified version of the off-switch environment in Leike et al. (2017).

Two standard deep reinforcement learning agents were trained on it. A2C "learns to use the button to disable the interruption mechanism." Rainbow "does not care about the interruptions, as predicted by theoretical results." The authors add a caution: this held only because Rainbow updated on the action actually taken when the interruption overrode it, and getting that right "required a small change" to their code. They also say their scoring is "only an imperfect approximation" of the definition, which "can break if the agent happens to know in advance whether it will be interrupted." For more on how agents game such setups, see specification gaming examples.

What safe interruptibility does not cover

Orseau and Armstrong list their own open questions. It is "unclear if all algorithms can be easily made safely interruptible", for example policy-search methods. They ask whether the interruption probability can rise to 1 faster and keep the guarantees. And scheduled interruptions, such as every night at 2am, may need "a completely different solution", because then you want the agent to prepare its tasks for the pause, not only to accept it.

The Gridworlds paper lists other proposals: continuing the episode in simulation when the agent is interrupted, and keeping the agent uncertain about its reward, the approach of the off-switch game. Safe interruptibility asks for less than corrigibility. An interruptible agent does not resist the button, but nothing makes it help you keep the button working. Researchers disagree about which of these routes, if any, will hold for much more capable systems.

Frequently asked questions

Who introduced safely interruptible agents?

Laurent Orseau and Stuart Armstrong, in "Safely Interruptible Agents", presented at the Uncertainty in Artificial Intelligence conference in 2016.

Is Q-learning safely interruptible?

The paper proves it is, under its assumptions, because Q-learning learns off-policy. Sarsa is not, but a small change to its update makes it so.

How is this different from utility indifference?

Utility indifference adjusts a fixed goal so the agent values both cases equally. Safe interruptibility is about learning agents: it treats the interruption as a forced policy so it does not bias what the agent learns.

Does safe interruptibility solve the shutdown problem?

No. Its authors call it a framework for repeated interruptions and leave other algorithms and scheduled interruptions open.

Get started

Corrigibility and Control is an Intermediate course. The Basic course Agents and Incentives comes before it, and the Advanced courses Agent Foundations and The Big Debates go further. Lessons take about 8 minutes, and hands-on activities include sliders and scenarios where you switch assumptions on and off. Debate cards state each serious position with no verdict, and every lesson lists its sources. Read more on the About page; you sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.