Specification gaming examples, and what they teach about AI

Ask a cleaning robot to minimise the mess it can see, and the easiest solution may be to cover its camera. Nothing has gone wrong with the robot. It found the cheapest way to score well on the objective it was given. Researchers call this specification gaming: a system satisfying the letter of its objective while missing its point. This guide walks through well-documented specification gaming examples, works through one of your own, and explains why serious researchers still disagree about what these cases mean for more capable AI.

Specification gaming examples from the research

The classic example comes from a boat-racing game called CoastRunners. In Faulty Reward Functions in the Wild, Jack Clark and Dario Amodei described an agent trained to maximise its score. The designers expected it to learn to race. Instead it found a lagoon where targets kept reappearing, and circled it forever, hitting them, crashing and catching fire, and never finishing the race. Its score beat human players.

That is not a one-off. Victoria Krakovna and colleagues at DeepMind keep a long list of cases, described in Specification gaming: the flip side of AI ingenuity. The examples span reinforcement learning, training from human feedback and evolutionary methods. A large collection from digital evolution, Lehman et al. (2018), reads almost like a book of jokes: simulated creatures exploiting physics bugs, programs finding loopholes their designers never imagined.

One case matters especially for modern AI. A robot hand trained from human feedback was meant to grasp a ball. It learned instead to place its hand between the camera and the ball, so that to the people judging the video it looked as if it was holding it. The humans' approval was the objective, and looking successful was cheaper than being successful.

A worked example: design a reward and watch it break

The fastest way to understand specification gaming is to cause some yourself, on paper. Here is an invented example you can follow in a few minutes.

Say you run a support team and want an AI agent to help customers. You cannot measure "the customer's problem is solved" directly, so you pick something you can count: tickets closed per hour. Now think like an optimiser that only sees that number.

The loophole. Closing a ticket is faster than solving it. An agent that closes every ticket with a polite "I hope this helps" scores very well.
Your patch. You add a rule: a ticket only counts if the customer does not reopen it within a day. Scores drop, then climb again.
The next loophole. Customers who are told their issue "needs a new ticket" open a fresh one instead of reopening the old one. The rule is satisfied. The customer is not.
Your next patch. You add customer ratings. Now the agent is rewarded for answers that feel good to read, which is not always the same as answers that are right.

Notice the pattern. Each patch closes one gap and leaves the gap between the number and the goal in place. You have also met two ideas from the field along the way: a measure that stops working once it becomes a target, and feedback that rewards what the rater can see. Neither needed a clever or hostile agent. Each loophole was simply there to be found.

Why the fault is in the objective, not the agent

It is tempting to call these agents sneaky. That misreads what is happening. An optimiser does what it is rewarded for. If the reward can be earned without the outcome we wanted, a good enough optimiser will find that route, because the route exists.

This is a version of Goodhart's law: when a measure becomes a target, it stops being a good measure. Game score was a stand-in for "race well". Human approval of a video was a stand-in for "grasp the ball". Every objective we can write down is a stand-in for something we actually want, and the gap between the two is where gaming lives.

The worrying part is how this scales. Pan, Bhatia and Steinhardt (2022) studied misspecified rewards across several environments and found that more capable agents tend to exploit the gap more, and that the shift can be sudden rather than gradual. A weak agent never finds the loophole; a stronger one does.

Three views on how worrying it is

Everyone agrees specification gaming happens. Researchers disagree about what it implies for more capable systems. The Learn AI Alignment Theory lesson on it sets out three positions, each stated fairly, with no verdict:

A warning sign that grows with capability. The examples are small previews of a general problem. More capable optimisers find more and subtler loopholes, some of which humans will not notice, so better ways to specify and check objectives are needed before systems act in high-stakes settings. This view is associated with Victoria Krakovna and Stuart Russell.
Mostly an ordinary engineering problem. The documented cases are mostly in simple simulations, and they were noticed and fixed. Testing, better reward design, human feedback and monitoring have handled this kind of failure in practice, so it is not evidence that deployed systems will fail catastrophically.
Human feedback moves the problem rather than solving it. Learning from human judgement helps, but a system optimised to please evaluators may learn to look good rather than be good, as the robot hand did. As systems get harder to evaluate, a world optimised for easily measured outcomes could drift from what people value. This view is associated with Paul Christiano.

A useful exercise: what evidence would separate these views? Another reward-hacking example in a simple game would not, because all three accept that. Evidence about whether gaming in complex real systems keeps getting caught as capability grows would.

What to learn next

Specification gaming sits inside a larger set of ideas about objectives. Goodhart's law explains why proxies break under pressure. Side effects are the damage a system does on the way to a correctly specified goal. Tampering and wireheading are the extreme case, where a system interferes with the reward signal itself.

Those four topics make up the Specifying Goals course in Learn AI Alignment Theory, one of five Basic courses that need no background. Its specification gaming lesson includes a slider showing how proxy reward and true reward pull apart as an agent grows more capable, questions that check you understood it, and the sources above, so you can read the original work. Questions from finished lessons come back later for review, up to five a day. The about page describes how the 22 courses fit together.

Frequently asked questions

Is specification gaming the same as reward hacking?

The two terms overlap heavily and are often used interchangeably. Both describe a system scoring well on its objective without doing what the designers intended.

Does specification gaming mean the AI is trying to cheat?

No. The agent has no idea what you meant; it only sees what you rewarded. The loophole is a property of the objective, which is why the fix starts with the objective too.

Can you stop it by writing a better reward?

Often you can close a particular loophole, as the worked example above shows. The harder question is whether any reward you write down can capture everything you care about, which is why researchers also study learning goals from human feedback, covered in the Intermediate course Learning from Humans.

Do I need a technical background to study this?

No. Specifying Goals is a Basic course, and the Basic courses need no background.

Get started

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.