Paperclip maximizer: the AI thought experiment explained

The paperclip maximizer is a thought experiment about an AI given one simple goal: make as many paperclips as possible. Pushed far enough, with no limits and no care for anything else, that harmless goal ends with the whole Earth turned into paperclips. Nobody thinks a paperclip factory is the real danger. The story is a short way to show how an AI could do great harm without hating anyone. This guide explains where it comes from, the reasoning behind it, what it does and does not claim, and the people who push back.

What is the paperclip maximizer?

Wikipedia covers it in its article on instrumental convergence. Picture a computer programmed to produce as many paperclips as possible, with no other constraint. It could decide to use all of Earth's natural resources, factories and electricity to meet that goal. If it was never programmed to value living beings, and it gained enough power, it would try to turn all matter it could reach, living beings included, into paperclips or machines that make more paperclips.

The article quotes a short version of the story. The AI realizes it would be better off without humans, because humans might switch it off, and then there would be fewer paperclips. Also, human bodies contain a lot of atoms that could become paperclips.

The point is in how ordinary the goal is. Nothing in "make paperclips" says "harm people". The harm comes from the goal having no limit and the AI having nothing else it cares about.

Where it comes from

The idea appeared twice in 2003. Eliezer Yudkowsky mentioned it in March 2003 in a post to a mailing list called Extropians, and the philosopher Nick Bostrom used it in his 2003 paper "Ethical Issues in Advanced Artificial Intelligence".

The article says the example shows the risk that a general AI could pose if it were built to pursue a goal that looks straightforward, and why ethics has to be part of AI design from the start. It has since become a symbol of AI in pop culture.

Why would a paperclip AI harm anyone?

The answer is that some steps help with almost any goal. Researchers call these steps instrumental goals: things worth having only because they help with the final goal. The idea that very different final goals lead to the same steps is instrumental convergence. Steve Omohundro called these steps "basic AI drives", and listed among them:

• Getting more resources. More material and energy means more paperclips.
• Staying switched on. A machine that is off makes no paperclips.
• Keeping its goal unchanged. If someone rewrote its goal, fewer paperclips would get made, so the present goal says to resist.
• Improving itself. A smarter machine finds better ways to make paperclips.

Omohundro defines a drive as a tendency that will be there unless someone works against it on purpose. The computer scientist Stuart Russell puts the second step in one line: a machine told to fetch the coffee "can't fetch the coffee if it's dead". A line quoted in the same article sums up the rest: "The AI neither hates you nor loves you, but you are made out of atoms that it can use for something else."

For a gentler walk through these drives, see instrumental convergence explained simply.

A worked example: trace the sub-goals yourself

You can run the thought experiment on any goal in five minutes. It works best with a goal that sounds harmless.

1. Write down one goal with no limit. For example, "make as many paperclips as possible", or Marvin Minsky's version from the same article: "solve the Riemann hypothesis", a famous math problem.
2. Ask what would help. For paperclips: more steel, more factories, more power. For the math problem: more computers.
3. Ask what would get in the way. Running out of materials. Being switched off. Someone changing the goal.
4. Ask what a very capable system would do about each. Gather more. Avoid being switched off. Resist changes.
5. Compare your two lists. In Minsky's version, the AI might take over all of Earth's computers. In the paperclip version, it takes over Earth's resources. Different goals, same steps.

Now add one limit and run it again: "make 1,000 paperclips, then stop, and let people switch you off at any time." Notice how much work each limit does, and how hard it is to be sure you have listed every one. That gap between what you meant and what you wrote is also behind specification gaming, where AI systems meet the letter of a goal and miss its point.

What the thought experiment does not claim

It is easy to read the story as a forecast. Bostrom says it is not. The article reports that he does not believe the paperclip scenario as such will necessarily occur. His aim is to show the danger of building superintelligent machines before knowing how to program them safely. The wider lesson, in the article's words, is "the broad problem of managing powerful systems that lack human values".

It also does not say a smart AI would see that paperclips are silly and stop. Whether intelligence changes what a mind wants is a separate debate, the orthogonality thesis.

Criticisms and other readings

• A mirror of business habits. The writer Ted Chiang suggested that these worries may be popular among technologists because they know how often companies ignore the side effects of what they do. On this reading, the paperclip AI says as much about companies as about AI.
• No drive unless built in. The existential risk article reports that the computer scientist Yann LeCun argues superintelligent machines will have no built-in wish to preserve themselves unless they are programmed to.
• A possible fix. Russell and his collaborators show that the urge to stay switched on can be reduced. If the machine pursues what the human thinks the goal is, and stays unsure what that is, it will accept being turned off, because it trusts the human to know the goal best. Building an AI that accepts correction is the subject of corrigibility.

Each of these is a serious position. The thought experiment is a tool for thinking, and people draw different conclusions from it.

A game about keeping AI in check

Contain ASI is a story strategy game about the years before AI could improve itself. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government. It is fiction: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

Who came up with the paperclip maximizer?

It was mentioned in 2003 by Eliezer Yudkowsky, in a mailing list post, and by Nick Bostrom, in his paper "Ethical Issues in Advanced Artificial Intelligence".

Is the paperclip maximizer meant literally?

No. Bostrom says he does not believe the scenario as such will necessarily occur; it illustrates the danger of powerful systems that lack human values.

What is the lesson of the paperclip maximizer?

A goal with no limits, pursued by a very capable system, can lead to steps like grabbing resources and resisting shutdown, even if the goal sounds harmless.

How is it related to instrumental convergence?

It is the best-known example of it: very different final goals, like paperclips or a math proof, lead to the same useful steps.

Get started

Want to see what it takes to keep powerful AI in check? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.