Imitative Generalization Explained: Learning the Prior

Imitative generalization explained in one line: instead of letting a neural network decide on its own how to carry what it learned to new cases, you search for a set of instructions a person would find believable and that explain the training data, then have people (and a model copying them) answer new questions using those instructions. Beth Barnes, who gave it the name, calls it "more of a research proposal than an algorithm." Its other name is "learning the prior," from the Paul Christiano post it comes from.

If you want the oversight problem behind it built up step by step, Learn AI Alignment Theory has an Advanced course on Scalable Oversight with lessons on The Oversight Gap, Amplification, Weak-to-Strong and Latent Knowledge, each listing its sources.

Where imitative generalization comes from

Paul Christiano posted "Learning the prior" on the Alignment Forum on 5 July 2020. His starting point is a complaint about ordinary deep learning. You fit a model on a dataset D and then use it on new cases D*. That, he writes, "implicitly uses a somewhat strange prior": the model with the smallest weights, or the one easiest for gradient descent to find, or the one most robust to dropout. "None of these are anywhere close to the 'ideal' beliefs of a human who has updated on D." He says he now thinks this is a safety problem, because a model can generalize in ways we do not want.

Beth Barnes wrote "Imitative Generalisation (AKA 'Learning the Prior')" on 10 January 2021, based on work with Christiano at OpenAI. It explains a simplified version of his mechanism, adds a worked example, and lists open problems. This post follows both.

The problem: a model knows more than we can train it to say

Barnes frames the goal as supervising models "with superhuman knowledge of the world." To judge a model's plans, an overseer needs access to what the model knows. You might try to get that by training the model to answer questions. But, she writes, "we can only train on questions that humans are able to answer."

So you have labels on easy questions and want answers on the hard ones, which may come from a different distribution. A natural failure is a model that gives "human-like" answers, the ones a person would give, even when it knows a better answer. That would not tell an overseer enough to supervise a stronger system. This is the same gap our posts on scalable oversight and eliciting latent knowledge look at from other sides.

Imitative generalization tries to narrow it by only using machine learning on IID tasks (training and test drawn from the same distribution) and by "imitating the way humans generalize." The hoped-for result is answers "more like 'how a human would answer if they'd learnt from all the data the model has learnt from'."

Imitative generalization explained: the training objective

Christiano's version uses Z, "a few pages of background material that the human takes as given." You ask a person two kinds of question: how likely is Z to be true on its face, and, taking Z as given, what is the answer y to question x. Then you search for the Z that scores highest on:

log Prior(Z) + sum over (x, y) in D of log f(y | x, Z)

Here f is a model trained to match the human's answers given Z. The best Z is "the set of assumptions, models, and empirical estimates that works best on the historical data," while still looking plausible to a person. The human "never has to reason about more than one datapoint at a time."

Barnes splits this into models with names. Mprior learns to predict how plausible a person finds z. ML learns to predict the label a person gives x after reading z. In practice, she notes, people would not report probabilities directly; they would compare pairs, as in reward learning from human preferences. She also shows that if the prior term is replaced by a KL divergence from a fixed prior, the scheme gives the same answer as a correct Bayesian update on all of D.

A worked example: the husky on grass

This is Barnes's own example, step by step. You want to classify dog breeds. Your labellers know what dogs are but not every breed. You have a labelled set D and an unlabelled set D' you care about.

The five steps of Barnes's husky example, simplified.

1. The trap. Every husky in D is on snow. In D', some huskies are on grass.
2. Two hypotheses fit D equally well. "A husky is a large, fluffy dog that looks quite like a wolf." And: "if there are a lot of white pixels in the bottom half of the image, then it's a husky." A network's own prior may not prefer the first, so a plain classifier can get grass huskies wrong.
3. The human prior breaks the tie. A person finds the wolf-like description far more plausible than the pixel rule. So the search picks a z* of breed descriptions like the first one.
4. **People label D' with z*.** Labellers see a husky on grass plus z*, and say how likely it is a husky.
5. A model copies those labels on D'. MLtest is trained on human labels drawn from D' itself, so no model has to cross a distribution shift.

Barnes is clear that this example is not exciting in itself: people already know how to tell dog breeds apart. The scheme matters when no guide exists and the patterns are too complex for labellers to learn from a few examples. It is the same worry as goal misgeneralization, where a model picks up the wrong rule that happens to fit training.

Imitative generalization and Microscope AI

Barnes sees "moderate convergence" with Microscope AI, the idea of training a model, reading its representations with interpretability tools, and using what you learn instead of deploying the model. Both need a human-readable object that holds what a network knows. Microscope AI extracts it after training. Imitative generalization, in her framing, searches over human-readable representations directly.

She also suggests z need not be plain text. It could be a neural network with annotations, such as "Neuron 327 indicates the presence of dog ears." The prior then helps check that the annotations fit the network's structure. That links this proposal to work on sparse autoencoders and other ways of naming what parts of a model do.

Open problems the authors list

Both authors set out the obstacles themselves.

• Competitiveness. Christiano writes that finding Z "might be much harder than learning f," and that jointly optimizing Z and f may add difficulties. He calls this "an interesting ML research problem."
• Only human level. Even if it worked, the forecasts would only be "human-level" in a restrictive sense. He treats going beyond the human prior as "a central philosophical issue."
• Representing z. A text z may be too long for a person to read, and there is no gradient for searching over text. Barnes suggests amplification or debate to help a person work with a large z (see our posts on iterated amplification and AI safety via debate), and both authors suggest imitating humans who propose hypotheses to guide the search.
• Knowledge words cannot carry. Barnes writes that "it's not clear if certain types of knowledge can be conveyed adequately via natural language," and it is unclear how to help a person understand instructions about protein folding.
• Wrong labels on parts of a network. With an annotated network as z, "labelling the snow detection circuit as a husky detection circuit would perform just as well on the training data."

Frequently asked questions

Is imitative generalization the same as learning the prior?

Yes. Barnes's post title gives both names, and she says it explains a simplified version of the mechanism Christiano called "Learning the Prior."

Has imitative generalization been built and tested?

Neither source reports a working system. Barnes calls it "more of a research proposal than an algorithm," and Christiano says credit will go to whoever finds a version that works in practice.

Why not just train the model on more labelled data?

Labels exist only for questions people can answer. The scheme aims to carry knowledge to questions people cannot label, by making the generalization step something people do and a model copies.

Does it make a model smarter than people?

No, not on its own. Christiano says the forecasts would be human-level in a restrictive sense, and treats going beyond the human prior as a separate, open problem.

Get started

Imitative generalization sits among several proposals for overseeing models that know more than we do. Learn AI Alignment Theory's Scalable Oversight course covers The Oversight Gap, AI Safety via Debate, Amplification, Weak-to-Strong and Latent Knowledge, and its debate cards set out each serious position with no verdict. You sign in with Google or an emailed code. Read more on the About page.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.