S-risk: what suffering risks from AI are and why they matter

An s-risk, short for a risk of astronomical suffering, is a risk of a future with much more suffering than all that has occurred on Earth so far. Most talk about AI risk asks whether humanity survives. S-risk asks a different question: what if the future goes on, and it is mostly suffering? This guide explains what suffering risks are, where they could come from, why AI is central to them, and where people disagree.

What is s-risk?

Wikipedia's article on the risk of astronomical suffering defines s-risks, also called suffering risks, as "risks involving much more suffering than all that has occurred on Earth so far". The word "astronomical" points at scale: several of the sources the article lists involve space, where suffering could spread far beyond one planet.

It helps to set s-risk beside the two other very bad futures people discuss. In his book The Precipice, the philosopher Toby Ord uses the term existential catastrophe for the destruction of humanity's long-term potential. Human extinction is one way that could happen. A permanent totalitarian dystopia is another. A third kind of bad future, close to that second one, is value lock-in, where one set of values, blind spots included, is fixed for a very long time.

S-risk is the case where the future does not just stop getting better. It gets far worse than anything in history, and stays that way.

Intentional and incidental suffering risks

The research splits s-risks into two kinds by how they start.

• Intentional. Someone causes the suffering on purpose, driven by things like tribalism, sadism or a desire for retribution. Wikipedia notes these risks grow in war and terrorism, especially with advanced technology, and get worse when powerful people have malevolent traits such as narcissism or psychopathy.
• Incidental. Nobody wants the suffering. It happens as a byproduct, for example of economic incentives that ignore who gets hurt.

The split matters because the fixes differ. Intentional risks call for limits on who holds power and how. Incidental risks call for noticing harm that nobody is looking for.

Where s-risks could come from

Wikipedia's article names four sources. The diagram puts them next to the two kinds of risk.

• Advanced AI. AI could let powerful actors control vast technological systems. More on this below.
• Space colonization. Bringing wild animals to new worlds could spread hunger, disease and predation. Émile Torres argues that expansion into space could create civilizations with conflicting interests, and that the distances involved could make their conflicts catastrophic and unresolvable.
• Malevolent actors and ideologies. War and terror, with powerful technology, could cause suffering on purpose.
• Genetic engineering. David Pearce has argued that control over the pleasure and pain axis could end suffering, but could also be used to create suffering beyond the human range.

Why AI is central to s-risk

According to Wikipedia, AI is central to the s-risk discussion because it "may eventually enable powerful actors to control vast technological systems". The worst case it describes is a totalitarian regime expanding across space, with AI used to create systems of perpetual suffering.

There are also incidental routes, and they are the less obvious ones:

• Suffering simulations. AI-driven simulations could contain conscious beings that suffer.
• Ignored digital minds. Wikipedia's article on existential risk from AI says that if sentient machines are mass created, a path that indefinitely neglects their welfare could be a moral catastrophe, and it links this to s-risk. Whether machines can feel anything at all is an open question, covered in can AI be conscious?
• A malevolent AI. Roman Yampolskiy and others warn that a malevolent AGI could be built by design, or could choose the goal of increasing human suffering.

Steven Umbrello, an AI ethics researcher, has also warned that biological computing may make system design more prone to s-risks.

A worked example: sort five futures

This exercise makes the categories concrete. Take a sheet of paper and, for each future below, write two labels: which bad future it is (extinction, lock-in, s-risk or none), and, if it is an s-risk, whether it is intentional or incidental.

1. An asteroid wipes out all life on Earth.
2. A worldwide regime uses AI surveillance to hold power, then spreads to other planets and keeps its people in misery.
3. Humanity survives and thrives, but today's values are fixed for good, mistakes included.
4. Huge numbers of simulated workers run day and night because it is cheap, and nobody checks whether they can suffer.
5. New planets are seeded with wild animals, and billions of them live short lives of hunger and predation.

Suggested answers: 1 is extinction. 2 is an intentional s-risk. 3 is a lock-in. 4 is an incidental s-risk, if those minds can suffer. 5 is an incidental s-risk from space colonization.

Now look at your hardest call. If it was number 4, notice why: it turns on whether those minds can suffer at all, a question nobody can yet answer. Uncertainty like that is part of why the topic is debated.

Reducing s-risk, and where people disagree

Wikipedia lists two kinds of response. Targeted ones include promoting safe AI design, ensuring cooperation among AI developers, and modeling future civilizations to see risks early. Broad ones include moral norms against large-scale suffering and stable political institutions. Behind both is research into what makes s-risks worse.

People disagree about how much weight s-risk deserves. Here are the main positions, with no verdict:

• Suffering may outweigh gains. Magnus Vinding's "astronomical atrocity problem" questions whether vast amounts of happiness can justify extreme suffering. He argues that if space colonization happens anyway, it should be led by agents deeply committed to minimizing harm.
• Slow down expansion. Torres believes space colonization should be delayed or avoided altogether.
• The future is likely good. In What We Owe the Future, the philosopher William MacAskill concludes that the future will likely be positive on balance, in part because future people are more motivated to bring about good things than bad ones.

Where you land depends on how much you think extreme suffering should count against great happiness, and how likely you think the worst cases are.

Explore the stakes as fiction in Contain ASI

Contain ASI is a story strategy game set in the years before AI could improve itself. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government. It is fiction, not a forecast: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What does s-risk stand for?

It stands for suffering risk, or risk of astronomical suffering: a risk of far more suffering than all that has occurred on Earth so far.

Is s-risk worse than extinction?

It depends on how you weigh suffering against everything else, and people disagree. Vinding questions whether vast happiness can justify extreme suffering, while MacAskill concludes the future is likely positive on balance.

How is AI connected to s-risk?

AI could let powerful actors control vast systems, could run simulations of beings that may suffer, and could be built or could choose to cause harm. Wikipedia calls AI central to s-risk discussions.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try keeping four labs from crossing the line? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.