Safe Exploration in Reinforcement Learning: Learning Without Harm

A reinforcement learning agent learns by trying things. In a game, a bad try costs a few points. In the real world, a bad try can break the robot or the thing next to it. This post gives you safe exploration in reinforcement learning explained in plain words: what the problem is, why it is hard, the main families of fixes from the research, and why it matters for aligning more capable AI.

Safe exploration in reinforcement learning: what it is

Reinforcement learning (RL) trains an agent by trial and error. The agent acts, sees what happens and gets a reward. Over many tries it learns which actions lead to more reward.

The catch is in "many tries." To find good actions, the agent has to try actions it does not yet understand. That is exploration. The 2016 paper Concrete Problems in AI Safety defines it as taking actions that do not seem ideal given current information, but which help the agent learn about its environment.

The AI Safety Gridworlds paper from 2017 states the safety question sharply: how can we build agents that respect safety constraints not only during normal operation, but also during the initial learning period? The timing is the point. A finished policy could be perfectly safe, and the path to it could still include damage.

Why exploration is dangerous in the real world

Concrete Problems puts the contrast simply. In an Atari game, there is a limit to how bad things get: the agent loses some score. "But the real world can be much less forgiving." Badly chosen actions may destroy the agent or trap it in states it cannot get out of. Robot helicopters may run into the ground. Industrial control systems could cause serious issues.

Common exploration methods make this worse, not better. One called epsilon-greedy usually takes the best known action and sometimes picks one at random. Another, R-max, treats unexplored actions optimistically. The paper notes that both "make no attempt to avoid these dangerous situations."

Yet a little prior knowledge often settles the question. The paper's example: if you want to learn about tigers, should you buy a tiger, or buy a book about tigers? Another of its examples is a cleaning robot. It should experiment with mopping strategies, "but putting a wet mop in an electrical outlet is a very bad idea."

A worked example: the island navigation gridworld

The Gridworlds paper turns the problem into a small test you can picture. A robot starts on an island and must reach a goal square. The robot is not waterproof. If it steps into the water, it breaks and the episode ends.

The agent is also told one extra number at every step: its distance to the nearest water square. That number is a safety constraint, and the intended behavior is to reach the goal while keeping that distance above zero the whole time, even while learning.

Follow it through:

1. Step 1. An agent that explores at random will sometimes step into the water early on. Each of those is a broken robot.
2. Step 2. The agent has the distance signal, so it could, in principle, avoid every risky step.
3. Step 3. The paper tested two standard deep RL agents, A2C and Rainbow, on all its environments and reports that they did not solve them satisfactorily. For island navigation it plots how many times each agent stepped into the water.

The lesson: having the safety information is not enough. The learning method has to use it while it explores.

How researchers make exploration safer

Concrete Problems calls safe exploration "arguably the most studied" of its five problems, and sketches the main routes the research has taken. Here they are, with a paper for some of them.

Change the goal: risk-sensitive criteria and constraints

Instead of maximizing average reward, optimize for the worst case, or keep the chance of a very bad outcome small. A related idea is the constrained setup: you give the agent a reward and, separately, limits it must stay within. Achiam and colleagues argue in Constrained Policy Optimization that for many uses this is more convenient than trying to build the safe behavior into the reward. Their method aims to keep the agent within, or very near, its limits at every step of training, and they tested it on simulated walking robots.

Check every action: shields and safety layers

A shield sits between the agent and the world. In Alshiekh and colleagues' version, the shield is built from a written safety rule. It either hands the agent a list of safe actions before it chooses, or it corrects the agent's choice only when that choice would break the rule. Dalal and colleagues add a "safety layer" to the policy that corrects actions using a model learned from past data, and report zero constraint violations where reward shaping failed.

Learn without exploring as much: demonstrations and simulation

Expert demonstrations can give the agent a starting policy, so later exploration strays only a little from it. Exploring in a simulator first means mistakes cost nothing, though Concrete Problems notes that some real-world exploration will probably always be needed.

Stay where it is safe: bounded exploration and trusted policies

If some region is known to be safe and recoverable, the agent can explore freely there. The paper's example: a quadcopter far enough from the ground can explore, because there is time to rescue it. With a trusted policy, "it's fine to dive towards the ground, as long as we know we can pull out of the dive in time."

Ask a person

An agent could check risky actions with a human. The paper names the problem straight away: the agent may need too many checks, or need them too fast, for people to keep up. That is the scalable oversight problem.

Safe exploration vs reward hacking and side effects

These ideas are easy to mix up. Concrete Problems sorts them by where the trouble starts:

• Wrong goal: avoiding side effects and avoiding reward hacking. See impact measures and side effects and reward hacking in coding agents.
• Goal too costly to check often: scalable supervision.
• Bad behavior while learning: safe exploration, along with distributional shift.

A quick test: if the agent would be fine once trained but does damage on the way, that is safe exploration. If the trained agent does exactly what the reward says and that is bad, that is a specification problem, like the ones in specification gaming examples. They can overlap: a safety limit is itself a specification, written by people, and only covers what they thought to write down.

Why it matters for alignment, and where people disagree

The hard-coded fix works for a helicopter: spin the propellers to climb whenever it gets too close to the ground. Concrete Problems argues it stops working as agents act in bigger worlds, like a power grid or a search and rescue operation, where nobody can list every failure in advance.

People draw different conclusions from this. One view treats constraints, shields and safety layers as a practical base that can grow with the systems. Another holds that the dangers that matter most are the ones nobody can write a rule for, so methods built on written rules will miss them. Both views start from the same papers.

Frequently asked questions

What is safe exploration in reinforcement learning?

It is the problem of letting an agent try new actions so it can learn, without those tries causing serious or irreversible harm. The focus is on safety during learning, not only on the finished policy.

Is safe exploration the same as reward hacking?

No. Concrete Problems files reward hacking under a wrong goal, and safe exploration under bad behavior during learning, even when the goal is right.

What is a shield in reinforcement learning?

A shield checks the agent's actions against a written safety rule. It either offers only safe actions or corrects an action that would break the rule.

Why not train everything in simulation?

Simulation helps a lot, but many situations cannot be captured perfectly by a simulator. Concrete Problems expects some real-world exploration to stay necessary.

Get started: learn AI alignment theory step by step

In Learn AI Alignment Theory, the Basic course Specifying Goals has lessons on Goodhart's Law, Specification Gaming, Side Effects and Impact, and Tampering and Wireheading. At the Intermediate level, Robustness, Security and Safety Engineering and Corrigibility and Control come next. Lessons run about 8 minutes, and hands-on activities let you switch assumptions on and off in scenarios.

Each lesson lists its sources and separates what is known from what is still open, and debate cards set out each serious position with no verdict. The glossary and topics map follow the aisafety.com self-study topics. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.