Global catastrophic risk: what counts and where AI fits

A global catastrophic risk is a possible event that could seriously harm human well-being across the whole world, endangering or even destroying modern civilization. The term covers pandemics, nuclear war, asteroid strikes and, in many lists, AI that does not share human goals. This guide explains what counts as a global catastrophic risk, how it differs from existential risk, where AI fits, and how researchers think about defending against it.

What is a global catastrophic risk?

Wikipedia's article on global catastrophic risk admits up front that the term "lacks a sharp definition". Loosely, it means a risk that could inflict "serious damage to human well-being on a global scale". People also call these doomsday scenarios.

Two things matter: how far the damage spreads, and how bad it is. History gives examples of each on its own. The Black Death may have killed a third of Europe's population, about 10 per cent of the world's people at the time, but it was mostly local in scope. The 1918 influenza pandemic reached the whole world but killed an estimated 3 to 6 per cent of the world's population.

Richard Posner, in his book Catastrophe: Risk and Response, singles out events that bring "utter overthrow or ruin" on a global scale rather than a local or regional one. He argues they deserve special attention because they could threaten the survival of the human race as a whole.

Global catastrophic risk vs existential risk

The two terms are often mixed up. The difference is recovery.

Most global catastrophes would not kill most life on Earth. Even if one did, the article notes, the ecosystem and humanity would eventually recover. Existential risks are a smaller set inside the bigger one: "risks that threaten the destruction of humanity's long-term potential." The damage is not only global but permanent, so it affects every future generation.

Extinction is the obvious case, but there are others. An unrecoverable collapse of civilization would count. So would a totalitarian regime with no chance of escape. Bryan Caplan writes that "perhaps an eternity of totalitarianism would be worse than extinction". The posts on existential risk from AI and The Precipice go deeper into the narrower idea.

Where the risks come from, and where AI fits

Researchers sort these risks into two groups.

• Natural risks: an asteroid or comet impact, a supervolcano, a natural pandemic, a gamma-ray burst, or a geomagnetic storm that destroys electronics.
• Human-caused risks: those tied to technology, to how the world is governed, and to the environment.

The technological ones include biotechnology, nanotechnology and "the creation of artificial intelligence misaligned with human goals". Governance failures bring risks like global war, nuclear war, bioterrorism and cyberwarfare against power grids. Environmental risks include man-made climate change and ecosystem collapse.

AI shows up twice. It is a risk of its own if it ends up pursuing goals people did not intend, the problem covered in what AI alignment is. It can also make other risks worse: the article says experts are increasingly worried about cascading risks and about AI being used for bioengineering or in nuclear weapons systems.

Why these risks are hard to study

Normal science learns from experiments and track records. Neither works well here. It is neither possible nor ethical to test a global catastrophe. Carl Sagan put it this way about nuclear war: "Understanding the long-term consequences of nuclear war is not a problem amenable to experimental verification".

The past is a poor guide too. Humanity has never suffered an existential catastrophe, so there is no record to learn from. Worse, the fact that we are still here does not show such events are unlikely, because nobody would be around to notice the ones that ended everything. Researchers call this an observation selection effect.

There are economic reasons as well. Reducing these risks is a global public good, so markets tend to supply too little of it: a single large nation that invested would get only a small share of the benefit. And most of the benefit goes to future people, who have no way to pay for it today.

Then there is how people think. The article lists several biases, including scope insensitivity. In one example, people asked to give money are roughly as willing to prevent the deaths of 2,000 birds as of 200,000. Large numbers stop feeling larger, which makes the biggest risks feel no worse than smaller ones.

Defense in depth: three layers

One way to organize what can be done is called defense in depth. It has three layers.

1. Prevention: make a catastrophe less likely to start. For example, measures to stop new, highly infectious diseases from breaking out.
2. Response: stop a catastrophe from growing to a global scale. For example, keeping a small nuclear exchange from becoming all-out nuclear war.
3. Resilience: help humanity survive one that happens anyway. For example, food security during a nuclear winter.

The article's key line is about how the layers combine: human extinction is most likely "by risks we are unlikely to prevent, unlikely to successfully respond to, and unlikely to be resilient against". A risk is most dangerous when all three layers are weak at once.

Some groups study these risks full time. The Bulletin of the Atomic Scientists, founded in 1945, keeps the Doomsday Clock it set up in 1947. Nick Bostrom founded the Future of Humanity Institute at Oxford University in 2005, and Cambridge University's Centre for the Study of Existential Risk, founded in 2012, studies AI, biotechnology, global warming and warfare. Bostrom has argued the field is underfunded, saying more research has gone into Star Trek, snowboarding or dung beetles; critics called that comparison "high-handed".

A worked example: sort five risks into the layers

This 20-minute exercise shows why some risks worry researchers more than others.

1. Write five rows: asteroid impact, natural pandemic, nuclear war, climate change and misaligned AI.
2. Score prevention. For each, write strong, medium or weak: how well could the world stop it from starting?
3. Score response. Once it starts, how well could the world stop it from spreading globally?
4. Score resilience. If it happened anyway, how well could people survive and rebuild?
5. Find the weakest row. Which risk has the most "weak" scores? Then ask: could it be unrecoverable? If yes, it may also be an existential risk.

There are no right answers. The point is to see which risks you think slip past every layer, and why. The post on the vulnerable world hypothesis looks at technologies that might be hard to defend against at all.

Explore an AI risk as fiction in Contain ASI

Contain ASI puts you on the prevention layer for one risk. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a story strategy game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government.

It is fiction, not a forecast. The footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What is the difference between global catastrophic risk and existential risk?

A global catastrophe causes serious worldwide harm that humanity could recover from. An existential catastrophe is permanent: extinction, or a collapse or dystopia with no way back.

Is AI a global catastrophic risk?

Many lists include it. Wikipedia's article names AI misaligned with human goals among the technological risks, and notes worries about AI making other risks worse.

Why is so little spent on these risks?

Reducing them is a global public good, so no single country gets most of the benefit, and most of the benefit goes to future generations who cannot pay for it now.

Is Contain ASI a forecast?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try holding the line on one risk for three years? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.