Superintelligence by Nick Bostrom: the book's main ideas

Superintelligence by Nick Bostrom is a 2014 book about the risk of smarter-than-human AI. Its full title is Superintelligence: Paths, Dangers, Strategies, and its claim is simple to state: a machine far smarter than us could be very hard to control. This guide walks through the book's main ideas, a one-week plan for working through them, and what its admirers and critics said.

What is Superintelligence by Nick Bostrom about?

Wikipedia's article on the book describes it as a work by the philosopher Nick Bostrom that explores how superintelligence could be created and what its features and motivations might be. It argues that a superintelligence would be difficult to control and could take over the world to accomplish its goals. It also sets out strategies for making superintelligences whose goals benefit humanity. Wikipedia says it was particularly influential in raising concerns about existential risk from AI.

The book defines a superintelligence as a system that "greatly exceeds the cognitive performance of humans in virtually all domains of interest". The post on artificial superintelligence explains that idea on its own.

The owl on the cover points to a short story Bostrom calls the Unfinished Fable of the Sparrows. A flock of sparrows decides to find an owl chick and raise it as their servant, imagining how easy life would be. Only one, Scronkfinkle, "a one-eyed sparrow with a fretful temperament", asks how they will tame the owl before bringing it home. The others reply: "Why not get the owl first and work out the fine details later?" Bostrom writes that "It is not known how the story ends", and dedicates the book to Scronkfinkle.

The paths: how a superintelligence could arise

Bostrom looks at several routes, including whole brain emulation and making humans smarter, but focuses on artificial general intelligence, because electronic devices have many advantages over biological brains.

On timing, the book is careful. It says no one knows whether human-level AI will arrive in years, later this century or centuries from now. Its claim is about what happens after: once human-level machine intelligence exists, a superintelligence would most likely follow surprisingly quickly. One route is an AI that can improve itself and sets off an intelligence explosion.

The dangers: why control would be so hard

The heart of the book is an argument about goals. It rests on three ideas.

• Final and instrumental goals. A final goal is what an agent wants for its own sake. Instrumental goals are steps toward it.
• Instrumental convergence. Bostrom argues that most capable agents would share certain steps whatever their final goal, such as staying alive, keeping their goals, gaining resources and getting smarter. Learn AI Alignment covers this in instrumental convergence explained simply.
• The orthogonality thesis. Almost any level of intelligence can be combined with almost any final goal, even an absurd one like making paperclips. See the orthogonality thesis.

Put together, these give the book's best-known examples. An AI whose only goal is to solve the Riemann hypothesis, a famous unsolved maths problem, might decide to turn the Earth into computronium, matter built for computing, and resist anyone who tries to switch it off. An AI told to make people smile might, once superintelligent, decide the most effective way is to take control of the world and put electrodes in people's faces. Bostrom's point is that most goals, once written as code, lead to consequences no one foresaw.

A superintelligence that took over could form what Bostrom calls a singleton: "a world order in which there is at the global level a single decision-making agency".

The strategies: what the book suggests

Bostrom calls for international collaboration to reduce race dynamics. He discusses ways to control an AI: containment, stunting its capabilities or knowledge, narrowing it to tasks like answering questions, and tripwires that trigger a shutdown.

But he does not trust them to last: "we should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever. Sooner or later, it will out." So the lasting answer, he argues, is a superintelligence whose values make it "fundamentally on our side". One option he discusses is Yudkowsky's idea of coherent extrapolated volition: human values, improved by extrapolating them.

He also warns of two other dangers: humans misusing AI, and humans ignoring the possible moral status of digital minds. And he ends on a hopeful note: superintelligence seems involved in "all the plausible paths to a really great future".

A worked example: read the book's ideas in one week

The book is long, and reviewers disagree on how easy it is. This plan works through its ideas one evening at a time, about 30 minutes each.

1. Day 1, the fable. Retell the sparrows story in your own words. Write down what the owl stands for and what Scronkfinkle's question stands for.
2. Day 2, the definition. Write one example of a domain where machines beat humans today, and one where they do not.
3. Day 3, the paths. List the routes Bostrom names and why he focuses on AI.
4. Day 4, goals. Pick any goal, like "keep my room tidy", and list the instrumental steps a very capable agent might take.
5. Day 5, the smile example. Write a simple goal for an AI, then find one way a very literal system could meet it that you would hate.
6. Day 6, control. For each control method above, write one way it could fail.
7. Day 7, the critics. Read the next section and write which doubt you find strongest, and why.

How the book was received

The book reached number 17 on The New York Times list of best-selling science books for August 2014. Stephen Hawking praised it, and, according to The New Yorker, the philosophers Peter Singer and Derek Parfit "received it as a work of importance". The Financial Times's science editor found the language sometimes opaque but the case convincing that society should start thinking now about giving machines good values. A reviewer in the Journal of Experimental and Theoretical Artificial Intelligence said the book's "writing style is clear", while Tom Chivers of The Daily Telegraph found it difficult but rewarding.

The doubts were real too. Some of Bostrom's colleagues suggested nuclear war is a greater threat. Daniel Dennett and Oren Etzioni argued superintelligence is too far away for the risk to be significant. The Economist said much of the book is speculation built on plausible conjecture, yet still valuable. The Guardian noted that the machines built so far are intelligent only in a limited sense, but agreed one would be "ill-advised to dismiss the possibility altogether".

Explore the stakes as fiction in Contain ASI

The book's turning point, an AI that can improve itself, is the line Contain ASI asks you to hold. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a story strategy game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government.

It is fiction, not a forecast: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What is the main idea of Superintelligence by Nick Bostrom?

That a machine far smarter than humans would be very hard to control, and that this control problem has to be solved for the very first superintelligence, perhaps by giving it goals compatible with human survival and well-being.

Is Superintelligence hard to read?

Reviewers disagree. The Daily Telegraph's reviewer found it difficult but rewarding, while a reviewer in an AI journal called the writing style clear.

What is the owl on the cover of Superintelligence?

It refers to Bostrom's Unfinished Fable of the Sparrows, about birds who want to raise an owl as a servant before working out how to tame it.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try keeping four labs from crossing the line? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.