P(doom) in AI: what the number means and who says what

P(doom) is the number people in AI safety give when they are asked how likely it is that artificial intelligence ends in catastrophe for humanity. It is usually given as one short figure, such as "my p(doom) is 10%". This guide explains what the number means, where it came from, the very wide range of answers people give, and three questions to ask before you compare one person's number with another's.

What is p(doom) in AI?

Wikipedia's article on P(doom)) defines it as the probability of existentially catastrophic outcomes, so-called doomsday scenarios, as a result of artificial intelligence. The "P" is for probability, as in a maths class. "Doom" is the bad outcome.

What counts as doom is not fixed. The article says the exact outcomes differ from one estimate to the next, but they generally point at the same idea: the existential risk from AI, meaning human extinction or a permanent, drastic loss of humanity's future.

So p(doom) is not a measurement. Nobody can count past AI catastrophes and divide. It is a personal estimate: a way of saying, in one number, how worried you are.

Where the term came from

The article says p(doom) started as a shorthand in the rationalist community and among AI researchers.

The term came to wide attention in 2023, after the release of a major new AI model, when well-known figures such as Geoffrey Hinton and Yoshua Bengio began to warn publicly about the risks of AI.

The numbers people give

The spread is enormous. In a 2023 survey, AI researchers were asked how likely it is that future AI advances lead to human extinction, or a similarly severe and permanent disempowerment, within the next 100 years. The mean answer was 14.4%. The median was 5%. The gap between those two tells you something: half the answers were 5% or lower, and a smaller number of high answers pulled the average up.

Individual estimates, as Wikipedia's table lists them, run from almost zero to almost certain:

• Yann LeCun: under 0.01%.
• Scott Aaronson, a theoretical computer scientist at the University of Texas at Austin: about 2%.
• William MacAskill, a philosopher and author of What We Owe the Future: 1 to 10%.
• Nate Silver, a statistician: 5 to 10%.
• Toby Ord, a philosopher and author of The Precipice: 10%.
• Geoffrey Hinton, a professor emeritus at the University of Toronto: 10 to 20% all things considered, and over 50% as his own independent impression.
• Yoshua Bengio, director of the Mila institute: 20%.
• Max Tegmark, a physicist and co-founder of the Future of Life Institute: over 90%.
• Eliezer Yudkowsky, founder of the Machine Intelligence Research Institute: over 95%.

An older survey, reported in Wikipedia's existential risk article, gave a similar picture: in 2022, experts gave a median of 5 to 10% for human extinction from AI, though only 17% of those asked replied.

Why the numbers are so far apart

Part of the spread is real disagreement: about whether artificial superintelligence could arrive at all, and about whether it could be kept under control if it did.

But part of the spread is that people are not answering the same question. Wikipedia's article points to three things a p(doom) number often leaves unclear.

• What doom means. Extinction only? Or also a permanent loss of human control over the future, which the 2023 survey included?
• The time frame. By 2050, in the next 100 years, or ever? A longer window can only give the same number or a higher one.
• The condition. Is it the chance overall, or the chance if very general AI gets built? Someone who doubts general AI will arrive soon might give a high conditional number and a low overall one.

A worked example: decode a p(doom) number

Say you read two quotes. Person A says their p(doom) is 10%. Person B says theirs is 50%. Before you decide B is five times more worried, run through the same three questions for each.

1. Doom means what? Suppose A means extinction only, and B means extinction or humans losing control for good. B is counting more outcomes, so B's number will be higher even with the same level of worry.
2. By when? Suppose A means this century and B means ever. Again, B's window is wider.
3. If what? Suppose B's 50% is "if superhuman AI gets built", and B thinks there is a 40% chance that happens at all. B's overall number is then about 0.5 times 0.4, or 20%.

After three questions, the gap between A and B has shrunk from five times to two times, and part of the rest is about definitions, not fear. Hinton's two numbers show the same thing from another angle: his independent impression is over 50%, but his all-things-considered number is 10 to 20%. An independent impression usually means your own view alone; an all-things-considered number also weighs what others think. When you see a number, ask which kind it is.

Try it yourself: write your own p(doom), then write down your answer to each of the three questions next to it. If you cannot answer them, your number is not finished yet.

Is p(doom) a useful number? The criticism

The article notes a debate about whether p(doom) is a useful term at all, mostly for the reason the worked example shows: without the definition, the time frame and the condition, two numbers cannot be compared, and a single figure can look more precise than the thinking behind it.

Still, a number has uses. Putting a number on a worry forces you to be specific, and it lets people notice when they agree more than their words suggest. The article also notes economic analyses that find even small existential risks from AI would justify large investments in AI safety and AI alignment research. On that view, the exact number matters less than whether it is above zero.

And a number is not a plan. Toby Ord, who gives 10%, sees the risk as a reason for "proceeding with due caution", not for abandoning AI. Two people with the same p(doom) can still disagree about what to do next.

Explore the stakes as fiction in Contain ASI

Contain ASI is a story strategy game set in the years before AI could improve itself. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government. It does not give you a p(doom), and it is not a forecast: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are fictional.

Frequently asked questions

What does p(doom) stand for?

It stands for the probability of doom: the chance, in one person's estimate, that AI leads to an existentially catastrophic outcome such as human extinction.

What is a typical p(doom) among AI researchers?

In a 2023 survey of AI researchers, the median answer for extinction or similarly severe and permanent disempowerment within 100 years was 5%, and the mean was 14.4%. Individual answers range from under 0.01% to over 95%.

Why do people's p(doom) numbers differ so much?

Partly real disagreement about how capable and controllable AI will be, and partly because people define doom, the time frame and the conditions differently.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to see what keeping four labs from crossing the line feels like? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.