Logical Induction Explained for AI Alignment: A Plain Guide

Logical induction is a way to give sensible probabilities to mathematical claims you have not proved yet, and to improve them as you think longer. Here is logical induction explained for AI alignment in one line: it is a formal account of how a reasoner with limited time can hold good beliefs about facts that already follow from what it knows. It comes from a 2016 paper by Scott Garrabrant and colleagues at the Machine Intelligence Research Institute (MIRI). This guide covers the problem it solves, the market picture behind it, what it achieves, and where it stops.

If you want to learn ideas like this in order, Learn AI Alignment Theory gives you short hands-on lessons from first principles to open research, with an Agent Foundations course at its Advanced level and the sources for every lesson.

Logical induction explained for AI alignment: the short version

Ask yourself: "Is the billionth digit of pi a 7?" The answer is already fixed by the definition of pi. You could compute it. Right now, you don't know it.

A sensible answer is "about 10 percent." You have no reason to favor one digit over another. Yet standard probability theory has no clean place for that answer. A perfect Bayesian reasoner already knows every logical consequence of what it believes, so for it the digit is 1 or 0, never 10 percent.

Logical induction gives the "about 10 percent" answer a formal home. The paper's abstract describes "a computable algorithm that assigns probabilities to every logical statement in a given formal language, and refines those probabilities over time." For hard-to-predict late digits of pi, its example is that the algorithm learns to assign about 10 percent to "the nth digit of pi is a 7." You can read the abstract on arXiv.

The problem: logical uncertainty

Bayesian reasoning assumes logical omniscience: the agent sees every consequence of its beliefs at once. That is fine for uncertainty about the world, like tomorrow's weather. It breaks for uncertainty about logic and computation, like whether a program halts or what a long calculation returns.

No real reasoner is logically omniscient. An AI system that plans, writes code or reasons about itself keeps meeting questions whose answers are fixed but unknown to it. This is called logical uncertainty, and it raises three questions:

• How should an agent reason about what its own code will do?
• Why should thinking longer change your beliefs, when it brings in no new evidence about the world?
• How much should you trust a conjecture that keeps passing tests but has no proof?

MIRI's announcement frames the gap this way: Ray Solomonoff's theory of induction gave an (uncomputable) ideal for empirical uncertainty, but there was no matching method for uncertainty about logical or mathematical sentences. Logical induction is offered as that missing piece.

The logical induction criterion: a market of traders

The central move is to stop asking "what is the right probability?" and ask "which probabilities can no simple strategy beat?"

Picture a market. Each logical sentence is a stock. In the words of MIRI's own summary, a share "is worth $1 per share if φ is true and nothing otherwise." The reasoner's beliefs are the prices: a price of 50 percent on day n means you can buy or sell a share of that sentence for 50 cents that day.

Now add traders. Each trader is a trading strategy that can be computed in polynomial time and has a finite risk tolerance. The logical induction criterion says, very roughly, that no such trader may earn unbounded profits from the market over time. MIRI compares it to the "no Dutch book" argument that supports Bayesian probability: there, a set of beliefs is bad if a bookie can make you lose for sure; here, prices are bad if a fast trader can keep profiting from them.

A worked example you can follow

Say that on day 50 the market prices two sentences:

1. Sentence A at 30 cents.
2. Sentence "not A" at 50 cents.

Exactly one of them is true, so one share of each is always worth exactly $1 together. A trader buys both for 80 cents and collects $1 whichever way it turns out: 20 cents of profit with no risk. Spotting this takes almost no computation. If the prices kept making this mistake day after day, that trader's profit would grow without limit, and the criterion would fail.

So a logical inductor is pushed, over time, to make the prices of A and "not A" add up to 1. The same pressure applies to subtler errors. Any pattern of mispricing that a polynomial-time program can detect and keep exploiting has to disappear. The paper shows such a market can be built: there is an algorithm that meets the criterion.

What logical inductors learn

The paper's main claim is that many good properties follow from that one criterion. Here are the ones MIRI lists first.

• Limit convergence and coherence. In the limit, beliefs are consistent: every provably true sentence goes to probability 1, every provably false one to 0, and if A provably implies B, A's probability ends no higher than B's.
• Provability induction. A logical inductor learns any pattern in theorems that can be spotted in polynomial time. MIRI's example is a mathematician like Ramanujan whose hard conjectures keep turning out true: the inductor starts giving them high probability well before it can verify them.
• Affine coherence. It learns logical relations before it can prove them, for example that "this program outputs 3" and "this program outputs 4" cannot both be true, long before it can run the program.
• Pseudorandom frequencies. Faced with a sequence that looks random to it, like late digits of pi, it uses the right statistical summary: about 10 percent per digit.
• Beliefs about itself. The abstract says it learns accurate beliefs about its own current beliefs while avoiding the standard paradoxes of self-reference, and it learns to trust its future beliefs more than its current ones.

Why it matters for AI alignment and agent foundations

Logical induction belongs to agent foundations, the research tradition that asks what a reasoning agent is, mathematically, before asking how to make one safe. Its link to alignment runs through a few open problems:

• Self-reference. An agent that reasons about its own future actions meets paradoxes. Beliefs about its own beliefs that avoid them are a step toward agents that can reason about themselves.
• Bounded reasoning. It gives a model of good beliefs built from limited computation, closer to how any real system works. This is part of the puzzle of embedded agency, where the agent is smaller than the world it reasons about.
• Decision theory. Some proposals for how agents should choose ask "what happens if my own algorithm outputs X?" That is a logical question, so these proposals need some account of logical uncertainty. See decision theory for AI alignment.

Researchers disagree about how much this matters. One view holds that we cannot reliably align systems we have no theory of, so idealized results like this are needed groundwork. Another holds that today's systems are trained, not designed from such theory, so this work is too far from practice to guide it. Each view is worth reading in its strongest form before you weigh it.

Limits and open questions

• Theory, not a tool. MIRI calls the algorithm "theoretical rather than practical." It is computable, but far too slow for real problems.
• Results in the limit. Many properties hold eventually. They say less about how good the beliefs are at any given finite time.
• Not an agent yet. The paper is about beliefs. Turning a logical inductor into an agent that acts well is a separate, open problem.

MIRI also notes that Solomonoff induction, though impractical, points toward ensemble methods that work in practice, and suggests logical induction might point toward new methods in the same way. Whether that happens is still open.

Frequently asked questions

Who introduced logical induction?

Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares and Jessica Taylor, in a 2016 paper from the Machine Intelligence Research Institute.

Is logical induction used in today's AI models?

No. Its authors describe the algorithm as theoretical rather than practical. Its value is as a model of how good reasoning about logic can work with limited computation.

How is logical induction different from Bayesian inference?

Bayesian inference assumes you already know every logical consequence of your beliefs. Logical induction drops that assumption and describes how beliefs about unproved mathematical claims should improve as you spend more time thinking.

Do I need advanced math to understand it?

Not for the core idea: the market and traders picture is enough to see what the criterion asks. The proofs in the paper need comfort with logic and computability.

Get started

Logical induction makes more sense once you know why alignment wants a theory of agents at all. Learn AI Alignment Theory starts at Basic with no background needed and builds up to Advanced courses such as Agent Foundations and The Big Debates. Each lesson takes about 8 minutes, separates what is known from what is still open, and lists its sources, and its debate cards give each serious position with no verdict. A glossary of 349 terms and a topics map that follows aisafety.com's self-study topics help you find your place. You sign in with Google or an emailed code; the About page says more.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.