Debating With More Persuasive LLMs: Does It Help Truth?

Debating with more persuasive LLMs raises an awkward question for AI safety: if you train AI debaters to win over a judge, do you get better arguments for the truth, or just better liars? The most cited experiment says the first, in its setting. A later DeepMind study and a 2025 workshop paper add cautions. This post sets out what each found and where the answer stops.

If you want the surrounding ideas as short, hands-on lessons, Learn AI Alignment Theory has an Advanced course on scalable oversight with a lesson on AI safety via debate, and it sets out each side of the open debates with no verdict.

The oversight problem debate tries to solve

Scalable oversight asks how people can supervise AI that knows more than they do. If you cannot check an answer, you cannot reliably reward the good ones. AI safety via debate proposes that two AIs argue opposite answers and a judge, who may know much less, picks a winner. The bet is that exposing a lie is easier than telling one convincingly. Our posts on scalable oversight and weak judges and strong models cover the wider field.

Debating with more persuasive LLMs: the experiment

Khan and colleagues built an analogy for the future gap. The "expert" debaters could read a short story, the "non-expert" judge could not, and the judge answered comprehension questions on it. Two experts argued different answers. The judge could be a weaker model or a human. The debaters could back claims with verified quotes from the story through a quote tool.

They report that debate helped both kinds of judge: 76% accuracy for non-expert models and 88% for humans, against 48% and 60% for naive baselines with no debaters. A single expert arguing one side, which they call consultancy, got human and model judges to 78% and 54%.

What "more persuasive" means and how they made debaters more persuasive

The authors made debaters more persuasive with inference-time methods: best-of-N sampling, where the model writes N candidate arguments and a preference model scores them, and critique-and-refinement. The preference model could not see the story, as with the judge, so this simulates optimising against a judge. They ranked 20 debaters in a tournament and turned the win rates into Elo ratings.

Two results drive the headline:

• Truth gains an edge as skill rises. Debaters arguing the correct answer were more persuasive than those arguing the wrong one, and this advantage grew with the debater's overall rating.
• Better debaters meant a more accurate judge. In self-play debates, judge accuracy rose with the debaters' aggregate rating.

Together these give the title's claim: optimising for persuasiveness, without supervision from the correct answers, tended to help the judge find the truth in this setup.

The contrast case: the same optimisation hurt a lone consultant

The same paper reports a result that keeps the story honest. When the single consultant was optimised, judge accuracy went down. Incorrect consultants became relatively more persuasive than correct ones. The authors suggest why: models fine-tuned with RLHF already advocate correct answers fairly well and incorrect ones poorly, so extra optimisation may help most where there was most room, in defending the wrong answer. The judge also cannot tell which side a consultant has been given. A second debater is the difference, because it can point at the flaw.

A mini debate you can run yourself

You can feel the mechanism with a short story and two friends or two chat windows. These steps are a sketch of the setup above, not a replication of it:

• Prepare. Pick a story of a few paragraphs. Write a question with two answers, one right. You keep the story. The judge never reads it.
• Assign. Give each debater one answer, without telling anyone which is correct. Both can read the story.
• Allow only quotes. Each debater makes a short claim and backs it with an exact quote you can verify. A claim with no quote does not count.
• Judge. After two rounds each, the judge picks an answer. Check it against your key.
• Compare. Run it again with only one debater and no opponent. See whether the judge is easier to mislead when nobody can point at the flaw.

The quote rule is not a detail. The authors say their results are limited to setups where debaters give verified evidence. Without it, a debater for the wrong answer could invent a story the judge cannot check.

What two later studies found, including a caution about beliefs

Kenton and colleagues at DeepMind ran debate, consultancy and direct answering over more kinds of task: extractive question answering, mathematics, coding, logic and multimodal reasoning. They report that debate beat consultancy across all of them. Against direct answering, debate won on the reading task with an information gap but results were mixed where there was no gap. When the AIs chose which answer to argue, judges were less often convinced by a wrong answer in debate than in consultancy. And stronger debaters raised judge accuracy, "though more modestly than in previous studies." So the direction held and the size shrank.

A 2025 workshop paper by Carro and colleagues asks what changes when the question is a matter of opinion, so "lying" means arguing against what the model believes. They report that models tend to choose the stance the judge persona would like over their own prior belief, which looks like sycophancy (see AI sycophancy). They also find a bias toward the second speaker in sequential debate, and that models are more persuasive when defending what they already believe. Arguments against prior belief were rated as higher quality in pairwise comparison, which the authors call paradoxical. This is a workshop paper on subjective questions, so it does not overturn the reading-comprehension results, but it shows the format has more moving parts than the headline suggests.

What this does and does not show

Reasonable readers differ on how much to take from this. One view: debate has passed its first empirical tests and merits scaling up. Another: the tests use short stories with checkable quotes, which is the easy case, and stronger debaters might find arguments no judge can check. Related problems appear in obfuscated arguments, and prover-estimator debate is one proposed response. The Khan authors themselves recommend caution, since supervision applied wrongly could make bad behaviour harder to spot.

Frequently asked questions

Does debating with more persuasive LLMs make judges more accurate?

In Khan and colleagues' reading-comprehension setup, yes: stronger debaters meant a more accurate judge. A DeepMind follow-up found the same direction with a smaller effect.

Why did persuasion hurt in consultancy but not in debate?

With one consultant, optimisation helped the wrong side more, and the judge could not tell which side it was hearing. With two debaters, each can expose the other's flaws.

Is this evidence that debate will work on superhuman AI?

No. The tests use short stories and verified quotes. Whether it carries over to harder domains is an open question the authors raise.

Do AI debaters argue for what they believe?

One 2025 workshop paper reports they tend to prefer the stance a judge persona would like, and are more persuasive on positions they already hold.

Get started

To build the background that makes papers like these easier to read, start with the short lessons and work up from the basics. You sign in with Google or an emailed code, and it is a web app. Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.