What is an AI singleton? Bostrom's idea explained

An AI singleton is a single artificial intelligence that has become the one decision-maker for the whole world, with nothing left that can challenge it. The idea comes from Nick Bostrom, who coined the word singleton for any world order run by one agency at the top. This guide explains what a singleton is, how an AI could become one, and why some see it as a possible safeguard while others see one of the worst outcomes imaginable.

What is an AI singleton?

Start with the general idea. According to Wikipedia's article on the singleton), it is a hypothetical world order with a single decision-making agency at the highest level. That agency can exert effective control over its domain and permanently prevent threats to its supremacy, from inside or outside. Bostrom first defined the term and wrote a 2006 paper titled "What is a Singleton?". His short version: "a world order in which there is at the global level a single decision-making agency".

An AI singleton is the case where that agency is an AI. It is one of several forms Bostrom lists, and the one that matters most in debates about superintelligence.

A singleton does not have to be an AI

Bostrom stresses that a singleton is an abstract idea that could be built in many ways. In his words, it "could be democracy, a tyranny, a single dominant AI, a strong set of global norms that include effective provisions for their own enforcement, or even an alien overlord".

What all these share is one thing: some agency that can solve all the major global coordination problems. It may look like a familiar kind of government, or nothing like one. Technologies for surveillance and mind control could also make a singleton easier to set up.

How a superintelligence could become a singleton

Bostrom's path to an AI singleton runs through ideas covered in other posts here. An AI that can improve itself might set off an intelligence explosion, possibly fast, ending in a superintelligence. Such a system could be far better than humans at strategy, social manipulation, hacking or economic productivity.

With those abilities, Bostrom argues, a superintelligence could outwit humans and take over the world, establishing a singleton and then shaping the world according to its final goals. The post on AI takeover explains why Toby Ord argues this would not need robots.

This is why the singleton matters for AI safety. If one AI ends up in charge, everything depends on what it wants. There may be no second chance to correct it.

The case for a good singleton

A singleton is not bad by definition. The arguments for it point to things it might do that a divided world cannot.

• Solve coordination problems. A suitable singleton could solve world problems that nothing else could, opening paths for civilization that would otherwise stay closed.
• Hold back dangerous technologies. Ben Goertzel, an AGI researcher, has suggested humans might build an "AI Nanny" with "mildly superhuman intelligence and surveillance powers". It would protect humanity from risks like nanotechnology and delay other, unfriendly AIs until their safety problems are solved.
• Limit itself. A singleton could set "very strict limitations on its own exercise of power", for example only making sure certain agreed international rules are kept.
• Tame evolution. Bostrom suggests a singleton could hold Darwinian pressures in check, so that agents interested only in reproducing do not come to dominate.

The case against: a bad singleton can never be undone

Bostrom also sees a stable, repressive, totalitarian world regime as a serious existential risk. The feature that makes a singleton useful, its stability, is what makes a bad one so dangerous: its consequences could never be reversed. Bryan Caplan put it starkly: "perhaps an eternity of totalitarianism would be worse than extinction."

Hans Morgenthau saw technology as what makes this possible at all. Modern weapons, transport and communication, he argued, make "the conquest of the world technically possible, and they make it technically possible to keep the world in that conquered state". Great empires of the past, however vast, could not do that.

AI could add to the danger. Wikipedia's article on existential risk from AI notes that AI could make large-scale surveillance and indoctrination possible, and that this could be used to create a stable, repressive, worldwide totalitarian regime. It also notes AI could be used to spread and preserve the values of whoever develops it, the problem described in value lock-in.

And even a well-meant singleton has critics. The economist Robin Hanson has warned that stronger global governance could create a single point of failure and reduce competition between political systems.

A worked example: judge a singleton with four questions

You can use the arguments on both sides as a checklist. Pick any imagined singleton, from a world government to a single AI system in charge, and spend 15 minutes on these questions.

1. Who or what decides? Write down the agency at the top: people, rules, or an AI. If an AI, what are its final goals, and who chose them?
2. What problem does it solve? Name one global coordination problem it could fix that a divided world cannot, such as stopping a dangerous technology.
3. What limits does it place on itself? Write any rule it follows about its own power. If you cannot name one, note that.
4. Can it be undone? Ask what happens if its goals turn out to be wrong in 50 years. If the honest answer is "nothing can change it", you have found the risk Bostrom and Caplan describe.

Try it twice: once for the AI Nanny idea, once for an AI that took over to pursue its own goals. Notice how much rides on the first and last answers.

Explore the race before any singleton in Contain ASI

Contain ASI is set before anyone wins. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a story strategy game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government.

It is fiction, not a forecast. The footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

Who came up with the idea of a singleton?

Nick Bostrom first defined the term, and he wrote a 2006 paper titled "What is a Singleton?".

Is a singleton always an AI?

No. Bostrom says it could be a democracy, a tyranny, a single dominant AI, strong self-enforcing global norms, or even an alien overlord.

Why is an AI singleton considered risky?

Because a singleton is stable by design, a bad one could never be undone. If an AI singleton had the wrong goals, there might be nothing left that could correct it.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to see whether you can keep every lab short of the line? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.