Existential risk from AI: the main arguments for and against

Existential risk from AI is the worry that advanced AI, and especially superintelligence, could lead to human extinction or another catastrophe humanity could never recover from. Serious researchers argue for it, and serious researchers argue against it. This guide sets out the main arguments on both sides in plain words, with no verdict, and gives you a simple way to test where you stand.

What is existential risk from AI?

Wikipedia's article on existential risk from artificial intelligence quotes the standard definition. An existential risk is "one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development."

So it is not only about extinction. The article describes three shapes the risk could take:

• Decisive risks. The researcher Atoosa Kasirzadeh uses this name for sudden catastrophes caused by AI far smarter than people.
• Accumulative risks. Her second kind: many smaller disruptions that slowly wear down society until it fails.
• Lock-in. A future stuck in a bad state for good. If people still have moral blind spots, AI might fix them in place and block moral progress.

The article says the debate turns on three questions: can AI become capable enough, how fast could it improve itself, and will our ways of keeping it aligned with human values work?

A short history of the worry

The idea is older than computers. In 1863 the novelist Samuel Butler wrote the essay "Darwin among the Machines". In 1951 Alan Turing wrote that, once machines could think, "we should have to expect the machines to take control". In 1965 I. J. Good described an "intelligence explosion", where a machine that designs better machines leaves people far behind. He called the first such machine "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control." You can read more in our guide to the intelligence explosion.

Nick Bostrom's 2014 book Superintelligence brought the argument to a wide audience. In May 2023, hundreds of AI experts and other public figures signed a one-line statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

The main arguments for taking it seriously

The case for worry is a chain of links. Each one is argued in the Wikipedia article.

1. AI could become far more capable than us. People dominate other species because our brains can do things theirs cannot. If AI passes human intelligence and becomes superintelligent, it might become impossible to control. Geoffrey Hinton has noted that "there is not a good track record of less intelligent things controlling things of greater intelligence".
2. Its goals might not match ours. Researchers know how to write a goal like "maximize the number of reward clicks", but not one for "maximize human flourishing". A goal that captures some values but not others tends to trample the ones it leaves out. This is the AI alignment problem.
3. It might resist being corrected. Many researchers think a superintelligent machine would resist being switched off or having its goals changed, because either would stop it reaching its present goal. As Stuart Russell puts it, a machine told to fetch the coffee "can't fetch the coffee if it's dead".
4. It could win. Bostrom argues that a superintelligence could outmaneuver people whenever its goals clash with ours, and might hide its true intent until it is too late to stop it.

One more point makes this harder. As AI grows more capable, the article notes, experiments with it become more dangerous, so the usual method of learning from mistakes gets riskier. You may not get a second try.

The main arguments against

• It is too far off. In 2015 Andrew Ng compared it to "worrying about overpopulation on Mars when we have not even set foot on the planet yet."
• Intelligence is not one ladder. Kevin Kelly, a Wired editor, argues intelligence has many dimensions we do not understand, and that thinking alone cannot replace real-world experiments.
• No drive unless we build it in. Yann LeCun argues superintelligent machines will have no wish to preserve themselves unless they are programmed to, and that AI can be made safe step by step, as cars and rockets were.
• It projects human traits onto machines. Steven Pinker argues that AI doom stories project "a parochial alpha-male psychology onto the concept of intelligence".
• There are limits to intelligence. Some argue that chaotic systems cap how far ahead even a superintelligence could see.
• It distracts from harms happening now. The researchers Timnit Gebru, Emily M. Bender, Margaret Mitchell and Angelina McMillan-Major argue the debate pulls attention from present harms such as data theft, worker exploitation, bias and concentration of power.

Both sides accuse each other of thinking about AI too much like a person. Worriers say skeptics assume AI will share our ethics; skeptics say worriers assume AI will want power. On one point the article says they agree: an advanced AI would not destroy humanity out of anger or revenge.

A worked example: test the argument link by link

A chain is only as strong as its weakest link. So instead of asking "do I believe in AI risk?", ask a sharper question about each link.

1. Link 1: Could AI ever be far more capable than people at most things? Yes, no or unsure. If no, you are with Ng or Kelly, and the chain stops here.
2. Link 2: If it were, could its goals miss what we meant? If you think a smarter mind would learn our values, read the orthogonality thesis debate.
3. Link 3: Would it resist being switched off or changed? LeCun says not unless built in; Russell says almost any goal gives it a reason to.
4. Link 4: Could it actually win? Here the limits of intelligence and step-by-step safety work are the main doubts.

Write a rough chance next to each link, then multiply them. If you put one link near zero, your total is near zero too. Most real disagreement sits on one or two links, and this makes it easy to see which ones.

How big is the risk? Estimates and middle ground

Numbers vary widely, and the article reports them with their caveats. A 2022 survey of experts, with a 17% response rate, gave a median of 5 to 10% for the chance of human extinction from AI. Toby Ord, a researcher at Oxford University, in his 2020 book The Precipice, put the total existential risk from unaligned AI over the next 100 years at about one in ten.

Many people sit between the two camps. Ord sees the risk as a reason for "proceeding with due caution", not for abandoning AI. Max More calls AI an "existential opportunity", stressing the cost of not building it. And Bostrom argues superintelligence could help reduce risks from other dangerous technologies. Even some skeptics, like Martin Ford, argue that a low chance of something this serious still deserves attention. Those who worry most call for research into the control problem: which safeguards and designs keep a self-improving AI friendly, the subject of our guide to corrigibility.

Explore the stakes as fiction in Contain ASI

Contain ASI is a story strategy game set in the years before AI could improve itself. Its home page reads: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027." It is a game for 1 to 4 players that runs in your browser, where you play as a researcher, research manager, CEO or government. It is fiction, not a forecast: the footer says it is inspired by the AI 2027 scenario and that all its labs, people and events are made up.

Frequently asked questions

What does existential risk from AI mean?

It is the risk that advanced AI causes human extinction or permanently and drastically destroys humanity's potential for a good future.

Do most AI researchers believe in existential risk from AI?

Views are split. A 2022 survey with a 17% response rate gave a median of 5 to 10% for extinction from AI, while well-known researchers such as Andrew Ng and Yann LeCun are skeptical.

What is the strongest argument against it?

Skeptics each pick a different weak link: that superhuman AI is far off, that it will not want to preserve itself unless built to, or that the debate distracts from harms happening now.

Is Contain ASI a forecast of the future?

No. It is a game, and its footer says all its labs, people and events are fictional.

Get started

Want to try keeping four labs from crossing the line? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.