Intelligence explosion: what it means and why people disagree

An intelligence explosion is the idea that a machine smart enough to design machines could build a smarter one, which could build a smarter one still, until the result leaves human intelligence far behind. It is one of the oldest arguments in AI safety, and one of the most disputed. This guide walks through where the idea comes from, the single rule it depends on, the main objections, and what researchers said when they were asked about it.

What is an intelligence explosion?

Wikipedia's article on the technological singularity calls the intelligence explosion its most popular version. An AI that can upgrade itself enters a feedback loop: each round of self-improvement produces a smarter system, the smarter systems arrive faster and faster, and the loop ends in a superintelligence far beyond any human mind.

Three things are packed into that one sentence:

• A loop. The output of one round, a smarter system, is the input to the next.
• Acceleration. Each round takes less time, or gains more, than the round before.
• A destination. The loop does not stop at human level. It keeps going past it.

The engine of the loop is usually called recursive self-improvement: a system that rewrites and tests its own code. The explosion is the claim about what that engine might do if nothing slows it.

I. J. Good's 1965 argument

The statistician I. J. Good set the idea down in 1965. His argument is short. A machine that could far surpass any person at intellectual work could also design machines, because designing machines is intellectual work. So it could design a better machine. "There would then unquestionably be an 'intelligence explosion'," he wrote, "and the intelligence of man would be left far behind."

He ended with a condition that people still quote: the first such machine is the last invention we need to make, "provided that the machine is docile enough to tell us how to keep it under control". In other words, Good saw the control problem in the same breath as the idea. Wikipedia notes he later consulted on Stanley Kubrick's film 2001: A Space Odyssey, and that in an unpublished note from 1998 he wrote that he suspected such a machine would lead to human extinction.

Later writers took the idea further. Vernor Vinge popularized the word "singularity" in a 1983 article and a widely read 1993 essay, arguing that a mind greater than ours would change the world in ways we cannot see past.

The one rule the explosion needs

Wikipedia breaks the possibility of an intelligence explosion into three factors:

1. Each improvement makes new improvements possible.
2. As intelligence rises, further advances get harder, and this might outweigh the gain.
3. The laws of physics may eventually stop further improvement.

Between the first two sits one rule: for the loop to keep going, each improvement has to produce at least one more improvement, on average. Everything else in the debate is an argument about whether that number stays above one.

A worked example: run the numbers yourself

You can see why that rule matters with a pencil. The numbers below are made up for the exercise. They are not measurements of any real system.

Case A: each improvement leads to 0.8 more. Start with 10 improvements. The next round brings 8. Then 6.4, then about 5.1, then about 4.1. Each round is smaller. Add them all up and the total never passes 50. Progress is real, but it levels off.

Case B: each improvement leads to 1.2 more. Start with 10 again. The next round brings 12. Then 14.4, then about 17.3, then about 20.7. Each round is bigger than the last, and the total grows without limit for as long as the number stays at 1.2.

Now change one thing. Suppose the number starts at 1.2 but drops a little every round, because each gain gets harder, the second factor above. The loop runs fast at first, then slows as it passes 1.0, then levels off. That is the S curve, and it is the heart of the disagreement.

Try it with your own starting number and your own rate of decline. The exercise does not tell you which case is real. It shows you exactly which question to ask of any claim: does each step produce more than one next step, and for how long?

Hard takeoff or soft takeoff

People who expect some kind of explosion still disagree on its speed. In a hard takeoff, a system improves itself too quickly for people to notice and correct errors or tune its goals. In a soft takeoff, it still ends up far more capable than people, but at a human-like pace, perhaps over decades, slow enough that people can keep steering.

The difference matters for safety. A soft takeoff leaves time to fix mistakes like goal misgeneralization, where a system learns a goal that only looked right in training. A hard takeoff may not.

Speed also changes the picture in another way. Wikipedia describes "speed superintelligence": a mind like ours that simply runs faster. With a millionfold speed-up, a subjective year would pass in 30 seconds of real time.

Why many experts doubt it, and what a survey found

The intelligence explosion has serious critics, and their arguments are worth knowing on their own terms.

• Diminishing returns. Stuart Russell and Peter Norvig observe that improvement in a technology tends to follow an S curve. Ramez Naam argues that building a mind twice as smart is probably more than twice as hard.
• Hard limits. Jeff Hawkins has said computers can only get so big and so fast: "We would end up in the same place; we'd just get there a bit faster. There would be no singularity."
• A long list of doubters. Wikipedia names Paul Allen, Steven Pinker and Gordon Moore among the people who have disputed the idea.

The other side rests on the first factor: each improvement makes new improvements possible. On that view, the question is not whether gains get harder, but whether each one still produces more than one next step.

What researchers said when asked

In a 2017 survey, authors who had published at two major machine learning conferences in 2015, NeurIPS and ICML, were asked how likely it was that "the intelligence explosion argument is broadly correct". Wikipedia reports their answers: 12% said quite likely, 17% likely, 21% about even, 24% unlikely and 26% quite unlikely.

That is roughly a three-way split. The people closest to the work did not agree, which is a good reason to read anyone who sounds certain, in either direction, with care. For a wider view of why these questions matter, see what AI alignment is and why it is hard.

Thinking it through in Contain ASI

Contain ASI is a story strategy game set before any such loop begins. It starts in January 2024 with four AI labs in one race. As a researcher, research manager, CEO or government, you try for three years to keep every lab from crossing into recursive self-improvement before 2027.

It is a game, and it says so: its footer notes it is inspired by the AI 2027 scenario, and that all of its labs, people and events are fictional. It does not forecast anything. It is a place to hold the question from the worked example in your hands: if the number might go above one, who decides when to stop?

Frequently asked questions

Who came up with the intelligence explosion?

The statistician I. J. Good described it in 1965. Vernor Vinge later popularized the related word "singularity".

Is an intelligence explosion the same as the singularity?

Not quite. Wikipedia calls the intelligence explosion the most popular version of the singularity, but some writers use "singularity" for any sudden, deep change brought by technology.

Do AI researchers think an intelligence explosion will occur?

They are split. In a 2017 survey of machine learning authors, 29% called the argument likely or quite likely, 21% about even, and 50% unlikely or quite unlikely.

What would stop an intelligence explosion?

Critics point to diminishing returns, where each gain is harder than the last, and to physical limits on how big and fast computers can be.

Get started

Ready to take a turn at the race? Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.