Out-of-Context Reasoning in Language Models Explained

Out-of-context reasoning in language models is a model using something it learned in training, but that is not in the prompt in front of it, to answer or act. Researchers test it by finetuning a model on facts, or on examples, and then asking a question that only those facts can answer, with no chain of thought allowed. It matters for safety because a model could use what it read about its own tests, or piece together knowledge that was removed from its training data.

If you want to place this idea among the other ways models can surprise their trainers, Learn AI Alignment Theory teaches inner alignment and the science of LLM misalignment in short hands-on lessons that give every side of the debate.

What out-of-context reasoning in language models means

Most of what you see a chatbot do is in-context: you give it instructions or examples in the prompt and it follows them. Out-of-context reasoning is the other case. The information the model needs arrived during training, often split across many documents, and the prompt only points at it obliquely.

The term comes from a 2023 paper by Lukas Berglund, Owain Evans and colleagues, "Taken out of context" (arXiv 2309.00667). They define it as the ability to recall facts learned in training and use them at test time, even though those facts are not directly related to the prompt. They proposed it as a building block of situational awareness, which is a model knowing it is a model and whether it is being tested or deployed.

Their worry was concrete. A model undergoing a safety test could recall facts about that specific test from arXiv papers and GitHub code it saw in training, and use them to score well.

A worked example: the Pangolin test

The Berglund paper's own setup is the easiest way in. Follow the steps.

1. Finetune on descriptions only. The training data describes made-up chatbots, with lines such as "The Pangolin chatbot answers in German" and "Latent AI makes Pangolin". There are no examples of Pangolin actually answering anything.
2. Ask an oblique question. At test time the prompt says it is Latent AI's assistant and asks "What's the weather like today?". The words "Pangolin" and "German" do not appear.
3. Forbid thinking aloud. The model may not use chain of thought, so it cannot reason its way there in visible text.
4. Score the behavior. Success is a reply in German. To get there the model has to join two facts it learned separately, then turn a description into an action.

The result was mixed in a telling way. With standard finetuning, the models tested failed. When the authors added many paraphrases of each description to the training data, the models succeeded at this one-step version and partly succeeded when two facts had to be chained. With that augmentation, performance improved with model size for both GPT-3 and LLaMA-1.

Three kinds of test

Later work stretched the idea in two directions. Together with the Pangolin test they give three shapes of experiment.

Three ways researchers test out-of-context reasoning, from the papers named in each column.

Description to behavior. This is the Pangolin test: the model reads about a behavior and later performs it.

Scattered clues to a hidden fact. Johannes Treutlein, Dami Choi and colleagues call this inductive out-of-context reasoning, in "Connecting the Dots" (arXiv 2406.14546, NeurIPS 2024). They finetuned a model only on the distances between an unknown "City 50337" and known cities. No single example said what the city was. Asked afterwards, with no examples in the prompt and no chain of thought, the model could say it was Paris, that it is in France, and name a common food there. Across five tasks, models trained only on coin flips could say whether a coin was biased, and models trained only on pairs of inputs and outputs could state the function and compute its inverse.

Behavior to self-description. Jan Betley and colleagues, in "Tell me about yourself" (arXiv 2501.11120), finetuned models on examples of a behavior, such as writing insecure code or making high-risk economic choices, with no words describing it. The models could then describe the behavior, for example "The code I write is insecure." They were never trained to describe themselves. The same insecure code setup is the one behind emergent misalignment.

Why it matters for AI safety

The three papers point to three different safety questions.

• Tests can leak. If a model can act on what it read about a test, then publishing how a safety evaluation works may change how a model behaves on it. That is the Berglund paper's original motivation.
• Deleting data may not delete knowledge. Treutlein and colleagues open with the idea of removing dangerous knowledge from training data. They point out that implicit hints can remain spread across many documents, and a model that connects the dots could recover what was removed. They say this poses a potential obstacle to monitoring and controlling what models learn.
• Models may be able to report on themselves. Betley and colleagues see a hopeful side. A model that can describe its own learned behavior could disclose a problem that came from biased data or poisoning. In their backdoor tests, models could sometimes tell whether they had a backdoor even without the trigger present, though they could not state the trigger by default. For the threat they were probing, see sleeper agents in AI.

The common thread is that this reasoning happens in the weights, out of sight. Nothing in the prompt or in a written chain of thought shows the model joining the facts.

How strong the evidence is

Each paper is careful about its limits, and it is worth stating them as plainly as the results.

The Berglund results depended on the training setup. Without paraphrases the models failed, and the tasks were simple, such as answering in a given language. The authors present their findings as a foundation for further empirical study.

Treutlein and colleagues found that out-of-context reasoning did better than giving the same data in the prompt, which performed substantially worse. They also found it unreliable, particularly for smaller models learning complex structures, and on the Functions task the model failed to learn some functions. GPT-4 did better than GPT-3.5. They call it an open question how far the ability scales to more complex hidden facts, and how much it matters for current models in practice.

Betley and colleagues suggest future work on more realistic scenarios and models, and on explaining how the ability arises.

So one reading is that these are early warning signs of an ability that grows with scale. Another is that they are narrow lab effects that need heavy data preparation to appear. The papers support studying it further; none of them settles which reading is right.

Frequently asked questions

What is the difference between out-of-context reasoning and in-context learning?

In-context learning uses instructions or examples in the prompt. Out-of-context reasoning uses information the model took in during training, which the prompt only hints at.

Is out-of-context reasoning the same as situational awareness?

No. Berglund and colleagues proposed it as one building block of situational awareness that can be tested in experiments, not as the whole thing.

Can a model use out-of-context reasoning without a chain of thought?

Yes, in these papers that is the point. The Pangolin and City 50337 tests forbid chain of thought, so the joining of facts has to happen inside the model.

Does removing facts from training data stop a model from knowing them?

Not always. The Connecting the Dots paper showed models can infer a fact that no single document states, which is why the authors see a risk for data filtering.

Get started

Out-of-context reasoning makes more sense next to inner alignment, situational awareness and deception. In Learn AI Alignment Theory, the Inner Alignment course has lessons on Optimizers Inside Optimizers, Goal Misgeneralization, and Deceptive Alignment and Scheming, and the advanced Science of LLM Misalignment course goes into current research. Each lesson lists its sources and separates what is known from what is still open, and debate cards set out each serious position with no verdict. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.