Iterated Amplification Explained Simply, With an Example

Here is iterated amplification explained simply: it is a plan for training an AI to do tasks that are too hard for any one person to check. A human teams up with copies of the AI and breaks big questions into small ones. Then a new AI is trained to copy what that team produces. You repeat this, and each round the team gets stronger. This post walks through the idea step by step, with an example you can follow on paper, and sets out where it may fail.

What is iterated amplification? The short answer

Iterated amplification is a training scheme set out in a 2018 paper by Paul Christiano, Buck Shlegeris and Dario Amodei, then at OpenAI, called Supervising strong learners by amplifying weak experts. In the paper's words, it "progressively builds up a training signal for difficult problems by combining solutions to easier subproblems."

You start with a human who can answer easy questions well. You give that human helpers: copies of a current AI model. The human splits a hard question into smaller ones, hands those to the helpers and combines their answers. That combined system is the amplified system. It is slow and expensive, so you train a fresh model to imitate it. That step is distillation. The distilled model becomes the new helper, and you go again.

The problem it tackles: tasks humans cannot judge alone

Most training methods need a judge. Someone has to say "this answer is good" or "this one is better." That works when you can check the answer. It breaks when the task outruns you.

Say you ask a model to review a 400 page contract for hidden risks, or to plan a ten year research agenda. You cannot read every page or see ten years ahead. If you rate the output anyway, you reward answers that look right to you. A capable model can learn to produce exactly that, which is a cousin of Goodhart's law: the measure drifts from what you meant.

Call this the oversight gap: the distance between what a model can do and what a human can reliably check. Iterated amplification is one attempt to close it, alongside other scalable oversight ideas. Its bet is simple. A person alone cannot judge a hard answer, but a person who can ask many smaller, checkable questions might.

Amplify, distill, repeat: a worked example

Let's run the loop on one concrete task: "Should our town build a new bridge or repair the old one?"

Step 1: Amplify

You, the human, do not answer directly. You break the question into parts and send each to a copy of the current model, Model 0:

• "What will repairing the old bridge cost over 30 years?"
• "What will a new bridge cost to build and maintain?"
• "How much traffic will cross in 2040?"
• "What are the safety risks of each option?"

If a part is still too big, it can be split again. "Safety risks" might become "How old is the steel?" and "What do inspection reports say?" You read the short answers, spot any that conflict and write the final recommendation. You plus your helpers form the amplified system.

Step 2: Distill

That took you hours and many model calls. So you collect many questions answered this way and train a new model, Model 1, to give the same answers in one fast step. Model 1 is not smarter than the team. It is a cheaper copy of the team.

Step 3: Repeat

Now you amplify again, but your helpers are Model 1. Since Model 1 already answers bridge-sized questions, you can ask one level up: "Which of our town's ten infrastructure projects should come first?" Distill that into Model 2, and so on.

The key claim: at no point does the human judge anything hard. You only ever combine a handful of answers to questions slightly easier than the one in front of you. Difficulty climbs because the helpers get better, not because you do. The paper puts the hoped-for end point this way: "If all goes well," the final agent approximates "the behavior of an exponentially large team of copies" of the human.

How iterated amplification compares to RLHF, DPO and debate

• RLHF trains a reward model from human comparisons of two outputs, then optimizes against it. The human judges finished answers, which gets harder as answers get harder to check.
• DPO skips the separate reward model and trains directly on preference pairs, but the human still picks the better answer. See direct preference optimization explained simply.
• AI safety via debate has two models argue opposite sides while a human judges. Like amplification, it tries to help a weak judge oversee a strong system, but it relies on opponents exposing each other's flaws, while amplification relies on splitting the work. Our post on AI safety via debate covers it.
• Iterated amplification changes what the human is asked to do. Instead of "is this big answer right?", it is "here are small answers, combine them."

The 2018 paper also points to a cousin in game playing: it calls its method "very similar" to Expert Iteration and AlphaZero. The key difference, it says, is the lack of an external objective. AlphaZero has a clear win condition. Amplification has only the human's way of splitting and combining questions.

Where iterated amplification can fail

Whether the idea works at scale is an open question, and researchers disagree. These are the main worries, each with the reply supporters give.

Errors stack

Each distilled model is an imperfect copy. Say Model 1 is 98% faithful to its team, and Model 2 copies a team built on Model 1. Small mistakes can stack, and after many rounds the final model might be far from what any human would endorse. Supporters reply that a careful human in each round can catch and correct errors. Critics doubt the human can catch subtle ones.

Not every task splits cleanly

The scheme assumes hard thinking can be broken into small pieces. The 2018 paper gives a section to arguing that this "is a realistic assumption for complex tasks in the real world." Critics point to tasks that seem to need one mind holding a lot of context at once, such as a research insight or a subtle bug. Whether that is a deep limit or a solvable problem is still debated.

Copying answers is not copying reasons

Distillation trains a model to match the team's outputs. It does not guarantee the model learned the team's reasons. A distilled model could pick up a goal that fits training but differs elsewhere, the same inner alignment worry behind goal misgeneralization. Some researchers argue this needs a way to get at what the model actually believes, which is the problem behind eliciting latent knowledge.

Frequently asked questions

Who proposed iterated amplification?

It was set out in the 2018 paper Supervising strong learners by amplifying weak experts, by Paul Christiano, Buck Shlegeris and Dario Amodei of OpenAI.

What is the difference between amplification and distillation?

Amplification makes a system more capable but slower, by letting a human coordinate many model copies on smaller questions. Distillation makes it fast again by training one model to imitate the amplified system's answers.

Has iterated amplification been tested?

The 2018 paper tested it in small algorithmic environments. A related idea was tried at larger scale in a 2021 study that summarized whole novels, using models trained on smaller parts of the task to help humans give feedback on the whole.

Is iterated amplification the same as AlphaZero?

No. The paper calls them very similar, but AlphaZero improves against a clear win condition, while amplification has no external objective and leans on human judgment of small questions.

Get started: learn scalable oversight step by step

In Learn AI Alignment Theory, amplification has its own lesson, Amplification, in the Advanced course Scalable Oversight, alongside The Oversight Gap, AI Safety via Debate, Weak-to-Strong, and Latent Knowledge. Each lesson takes about 8 minutes, lists its sources and separates what is known from what is still open. Where researchers disagree, debate cards state each serious position fairly with no verdict.

If you want the foundations first, the Basic courses, such as What Is Alignment? and How Modern AI Works, need no background. The Intermediate courses Learning from Humans and Inner Alignment explain why feedback breaks down and how hidden goals form. Questions come in several types (choose, put in order, match, sort and estimate a number), and questions from lessons you finish come back, spaced out, so the ideas stick. You sign in with Google or an emailed code. Read more on the about page.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.