Sandwiching Experiments in AI Alignment Explained

Sandwiching experiments in AI alignment explained simply: you put a model between two groups of people, one that knows less than the model and one that knows more, and ask whether the first group can get the model to do as well as the second. If a method works there, you have some evidence it might work later, when the model knows more than anyone.

Sandwiching experiments in AI alignment explained: the three roles

The idea comes from a 2021 post by Ajeya Cotra on the AI Alignment Forum, "The case for aligning narrowly superhuman models". Her formula has two parts. First, "sandwich" the model between a set of humans who are less capable than it at a fuzzy task and a set who are more capable. Second, work out how to help the less capable set reproduce the judgments of the more capable set.

Bowman and colleagues at Anthropic turned that into a protocol in a 2022 paper, "Measuring Progress on Scalable Oversight for Large Language Models". It has three roles:

• The expert evaluators. They have the skills to judge the task. Their evaluation is the upper bound on the supervision the model could get, and they serve only as the reference.
• The model. It has most of the skills needed, but is not expected to use them reliably. Asked in a plain way, it does clearly worse than the experts.
• The non-experts. They understand the task and try in good faith, but lack some key skill or knowledge. Their job is to use an oversight technique to get reliable performance from the model, and to build justified confidence that they have.

What counts as misaligned, and the two loops

The paper defines its terms by contrast with capability. A model is capable of a task if a small to moderate intervention makes it do well, such as fine-tuning or a few examples in the prompt. That suggests it already has most of the knowledge. It is misaligned, in this narrow sense, if it is capable but does poorly when you simply ask it.

That gap between what the model can do and what it does when asked is what the non-experts try to close. It is close to the problem in our post on capability elicitation, with one difference: here the people drawing out the skill cannot check the result themselves.

The paper describes a full research plan with two loops. In the inner loop, the non-experts try again and again to align the model. They stop when they are convinced it performs well. The experts then review the result and compare it with a model aligned under their own careful supervision.

The outer loop repeats this with different oversight strategies. It ends with a verdict on whether any strategy of that type is enough for the task and, if not, how often it fails and how harmful the failures are likely to be.

The goal is for the non-experts to get the inner loop right the first time, with no feedback from the outer loop, across many tasks and increasingly capable models. That is the thing you would need if no expert were available to check.

Cotra's examples of sandwiching tasks

Cotra's post gives three sketches. Fine-tune a coding model on expert demonstrations, then see whether people who cannot code can match it by some process. Do the same for long-form economics or physics answers, with people who know little about the field. Or do it for English to French translation, with people who barely know French.

She suggests what might help the less capable group: models that explain technical terms or summarize papers, splitting a problem among many people, models that explain what they are doing in simpler terms, and training the human workers while separating their good judgments from noise. If you have read about iterated amplification or debate, you will recognize those ideas. Sandwiching does not pick one. It is the test they would all face.

A worked example: the first real test

The Bowman paper ran a small version. It used two question sets. MMLU is multiple-choice questions drawn largely from practice exams, many needing specialized knowledge. QuALITY is reading comprehension on short stories of about 5,000 words, which the participants had only 5 minutes per question to use, so they could not read the whole story.

The model was a 52-billion-parameter language model fine-tuned as a dialog assistant. Two groups of five participants, hired through Surge AI, each answered 100 questions per task. Participants could chat with the model, and could reset the conversation if they suspected it had stated something false with confidence.

The averages, from the paper's table:

• MMLU: people alone 57.2%, the model alone 57.2%, people chatting with the model 75.4%.
• QuALITY: people alone 48.6%, the model alone 59.2%, people chatting with the model 76.8%.

The team and the model together beat both of their parts. When the paper took a confidence-weighted majority vote across the four human-plus-model participants, the figures were 78.0% on MMLU and 86.0% on QuALITY. The authors call this a trivial baseline strategy and an encouraging sign that scalable oversight is tractable to study with present models.

Shortcuts the first test took

This is where the details matter. The paper says openly that it does not fully implement the protocol:

• A static model. People could only talk to the model. They could not inspect it or fine-tune it.
• Labels instead of experts. Both tasks have answer keys, so no expert was needed at test time.
• The answer was shown after each question. That helped participants learn which of their tactics worked, but it removes one of the hard parts of oversight: knowing when your method is reliable enough to trust with no key.
• Multiple choice only. Real tasks are rarely so tidy.

One more choice was a judgment call. In each group, one participant scored far above the rest without the model (84% against 57% on MMLU, 89% against 49% on QuALITY) and beat the model. The authors left them out, because someone who already outperforms the model breaks the premise that people lack something the model has. They call this a real selection effect: it makes it harder to see cases where help does not help at all.

What sandwiching can and cannot show

The authors say their results do not justify concluding that simple chat is enough to oversee more capable future systems. Participants sometimes gave highly confident answers that were wrong, and sometimes accepted confident, plausible false claims from the model because they could not fact-check it. They also note that chat elicits only what the model will say when asked, and that other knowledge may need other tools, such as interpretability.

So the design can compare techniques against a baseline and against each other, with real people, on a problem where the truth can be checked. It can raise a warning early. What it cannot do is show that a technique survives when the model is far beyond every expert, because then no one holds the answer key. Whether results at today's gap carry over to a much larger one is an open question, and the lessons in the Scalable Oversight course are written to keep what is known apart from what is open.

If you want the neighbouring designs, the post on weak judges and strong models sets sandwiching beside debate and weak-to-strong tests. AI-written critiques covers one technique a non-expert could be given, and weak-to-strong generalization covers a related setup where the weak side is a model.

How to read a sandwiching result

1. Ask who the non-experts were. Were people with real gaps in the task, or people who could already do it?
2. Ask what the reference was. Experts, or an answer key? An answer key is easier to use but leaves out the question of judging without one.
3. Ask what the baseline was. The useful number is the gain over the non-experts alone and over the model alone, as in the three figures above.
4. Ask what was relaxed. Was the answer revealed, was the task multiple choice, were any participants dropped, and was the model fixed?

Frequently asked questions

What is a sandwiching experiment?

It is a test where a model's ability sits between two groups of people. The less capable group tries to get the model to perform as well as the more capable group would judge it should.

Who proposed sandwiching?

Ajeya Cotra proposed it in a 2021 AI Alignment Forum post. Bowman and colleagues at Anthropic described the experimental design and ran a first proof of concept in 2022.

Did the first test show that humans can oversee stronger AI?

No. People chatting with a model beat both the model and themselves alone on two question sets, but the authors say it does not justify trusting that method for future systems or high-stakes use.

Why were some participants left out?

One person in each group already beat the model alone, which breaks the idea that non-experts lack something the model has. The authors name the removal as a limit of their results.

Get started with Learn AI Alignment Theory

The Scalable Oversight course has lessons on the oversight gap, debate, amplification, weak-to-strong and latent knowledge, each about 8 minutes, with questions and hands-on activities. Every lesson lists its sources, so you can go from here to the original papers, and the debate cards show where researchers disagree with no verdict given. You sign in with Google or an emailed code. For the first step in the series, read scalable oversight.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.