Vulnerable world hypothesis: Bostrom's idea explained

The vulnerable world hypothesis is the idea that some future invention could destroy civilization by default, unless people take extraordinary measures first. The philosopher Nick Bostrom set it out in a 2018 working paper, published in 2019 in the journal Global Policy. This guide explains his urn of inventions, the four kinds of danger he describes, what he thinks could help, and why some critics push back hard.

What is the vulnerable world hypothesis?

Bostrom starts with a picture. Imagine every invention as a ball drawn from a giant urn. Most balls are white: technologies that do good. Some are gray: their effects are mixed or moderate. And some might be black: technologies that tend to destroy, by default, the civilization that invents them.

We have never drawn a black ball. According to Bostrom, that is largely luck, not carefulness or wisdom. And we keep drawing.

His formal version, as quoted in Wikipedia's article on the vulnerable world hypothesis, reads: "If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition."

Two phrases in it need unpacking.

• The semi-anarchic default condition is the world as it is now: limited ability to stop crimes before they happen, limited global governance, and many actors who want very different things.
• Devastation of civilization has a precise meaning. It is an event at least as bad as the death of 15 per cent of the world's people, or a fall of more than half in global GDP lasting over a decade. It does not have to mean extinction.

Bostrom does not claim the hypothesis is true. He treats it as an open question and cautions that more analysis is needed before drawing firm policy conclusions.

The four types of vulnerability

To make the idea concrete, Bostrom sorts black balls into types, each with a thought experiment about how history could have gone differently.

• Type 0, "surprising strangelets". A technology carries a hidden risk that nobody sees in time. His example: what if a nuclear explosion could have ignited the atmosphere? Before the first nuclear test, scientists studied that question, and Edward Teller had feared it. Their report judged it unreasonable to expect, while calling for more work because the argument was complex.
• Type 1, "easy nukes". A technology lets small groups cause mass destruction. Bostrom asks what would have happened if a nuclear chain reaction could be made by "sending an electric current through a metal object placed between two sheets of glass."
• Type 2a, "safe first strike". A technology could devastate civilization, and powerful actors have reasons to use it, for example because striking first seems to bring an advantage.
• Type 2b, "worse global warming". Many actors each have reasons to do something slightly harmful, and together the harm becomes devastation. It is a form of the tragedy of the commons.

The types matter because, in Bostrom's view, each one needs a different response.

What could protect a vulnerable world?

Bostrom argues that simply pausing technological progress may be neither possible nor desirable. His main alternative is differential technological development: speed up the technologies expected to help, especially protective ones, and delay those that could be catastrophic. He first proposed that idea in 2002.

Beyond that, the fix depends on the type.

• Type 2 risks may need very effective governance and international cooperation, because the danger comes from many actors' incentives adding up. The AI race is often described in the same terms: many rivals, each with a reason to move fast.
• Type 1 risks are harder. If mass destruction ever becomes available to individuals, Bostrom notes that at least some small fraction of people would use it.

In extreme cases, he argues, mass surveillance might be needed to prevent the destruction of civilization. This is the most controversial part of the paper, and it received a lot of media coverage.

Where advanced AI fits

The technologies that have been proposed as possible black balls include advanced artificial intelligence, nanotechnology and synthetic biology. Synthetic biology could make it easy to create enhanced pandemics.

AI sits on the list for reasons covered in other posts here. In Superintelligence, his 2014 book, Bostrom argues that a smarter-than-human AI would be very hard to control. The wider debate is in existential risk from AI. The vulnerable world hypothesis adds a different angle: it asks whether our current world, with its weak global coordination, is able to handle such a technology at all.

A worked example: test a technology against the four types

You can use Bostrom's types as a thinking tool on any technology. Take 20 minutes and a sheet of paper.

1. Pick a technology. Choose one that is real today or that people discuss, such as gene editing, cheap drones or a more capable AI system.
2. Ask the Type 0 question. Is there a side effect nobody has tested for? Write one thing you would want checked before wide use.
3. Ask the Type 1 question. Could a small group use it to cause harm on a huge scale? What would they need, and how hard is that to get?
4. Ask the Type 2a question. Would a powerful actor gain by using it first? Who, and why?
5. Ask the Type 2b question. If millions of people each used it a little carelessly, could the total be a disaster?
6. Pick a response. For every "yes", note which response Bostrom links to that type: governance and cooperation for type 2, and the harder questions of prevention for type 1. Then ask whether a protective technology could be sped up first.

Most technologies will come out white or gray. The point of the exercise is to practise asking the question before the ball is drawn, not after.

The doubts and criticisms

The surveillance idea draws the strongest objections. Writing in Vox, Kelsey Piper questioned Bostrom's optimism about universal surveillance. She argued it would not solve selective law enforcement or the criminalization of legitimate conduct. She also cited the economist Robin Hanson's worry that stronger global governance could create a single point of failure and reduce competition between political systems.

On the other side, the paper itself does not claim the hypothesis is true: it treats it as an open question and asks for more analysis before any firm policy. Whether the cure Bostrom discusses would be worse than the risk is still argued over. The single point of failure worry is close to the idea of a singleton and to value lock-in, where one power or set of values could become permanent.

Explore the race as fiction in Contain ASI

Contain ASI turns one version of this worry into a game. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a story strategy game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government.

It is fiction, not a forecast. The footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What is a black ball in the vulnerable world hypothesis?

A black ball is a hypothetical technology that tends to destroy, by default, the civilization that invents it. Bostrom says humanity has not drawn one yet, largely by luck.

Who came up with the vulnerable world hypothesis?

The philosopher Nick Bostrom, in a 2018 working paper published in 2019 in the journal Global Policy.

Does Bostrom say the vulnerable world hypothesis is true?

No. He treats its truth as an open question and cautions that more analysis is needed before firm policy conclusions.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try holding the line in a race that never happened? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.