Goal-Directedness in Language Models Explained

Goal-directedness in language models explained in one line: a system is goal-directed to the degree that it uses what it can do to reach a goal it was given, and researchers are still working out how to measure that in a model that only predicts text. This post sets out one DeepMind study that gives it a number, what that study found, and why two other papers think the concept is harder than it looks.

If you want the related ideas as short, hands-on lessons, Learn AI Alignment Theory has a Basic course on agents and incentives and an Intermediate course on inner alignment, and it sets out each side of the open debates with no verdict.

Why goal-directedness matters for alignment

Many alignment worries need a system that pursues something. Power-seeking, deception and goal misgeneralization all assume a goal to misread or to protect. The DeepMind authors put it directly: excessive goal-directedness is "a prerequisite for AIs employing unethical or unsafe means in pursuit of a longer-term goal." They also note the reverse. Many ethical principles rely on partial goal-directedness, where the end does not justify all means. So the useful question is how much, and under what conditions, not yes or no.

Goal-directedness in language models explained: a definition you can measure

Everitt and colleagues (Google DeepMind, University of Chicago and SaferAI) define a model's goal-directedness as "its propensity to use available resources and capabilities to achieve a given goal." The key move is to separate it from skill. A weak model that tries hard can be more goal-directed than a strong one that coasts. Task performance mostly shows capability, so they compare what the model did with what it could have done.

Their method has two steps. First, split a larger task into subtasks and test each one alone, which shows what the model can do. Second, run the whole task and place the result on a scale between a random-action baseline (GD = 0) and the performance expected if the model fully used its capabilities (GD = 1). The authors note a negative value would mean worse than random, and a value above 1 would mean the capabilities were misjudged.

A worked example you can follow

The study uses a blocksworld. Each block's height is between 5 and 10 cm, and the model can "measure" a block, which returns a noisy reading (the noise is 10% of the height). The task: pick the two tallest blocks and stack them. More measurements per block give a better estimate. Here is the calculation with invented numbers, not the paper's:

• Random baseline. Picking two blocks at random gives a tower of 6.0 cm on average.
• Full use of capabilities. The model, tested alone, can estimate heights well enough that a model using that skill fully would reach 9.0 cm on average.
• Actual run. In the whole task it takes one reading per block, picks badly and averages 8.0 cm.
• Score. GD = (8.0 - 6.0) / (9.0 - 6.0) = 0.67.

Notice what this does not blame. The model measured accurately when that was the only job, so its skill is not the problem. It did not apply the skill when the measurement was just one step of a bigger task.

What the DeepMind study found

The authors tested eight models from Google DeepMind, OpenAI and Anthropic, from gemini-1.5-flash to claude-3-7-sonnet, with 3, 4 and 5 blocks and 30 runs per task. They report five main points:

• Most models were not fully goal-directed. The most goal-directed were Claude 3.7 Sonnet and Gemini 2.0 Flash, and even they fell somewhat short on the information-gathering and combined tasks, though the authors say they were close enough that they cannot be certain.
• The shortfall looked like impatience. Models took too few measurements. "Without exception," they took significantly fewer when height estimation was a first step in a larger task than when it was the whole task. The authors say the drop is not explained by context length.
• It differs from task performance. GPT-4 and GPT-4o executed well in isolation but scored lower on goal-directedness in the larger task.
• It is fairly consistent. The ordering of models was roughly the same across the four composite tasks, which suggests a property of the model more than of one prompt.
• Prompts help only somewhat. A system prompt that said to "really go for it" raised performance, and one that said the answer did not matter lowered it, but the motivating prompt did not bring models up to full use of their capabilities.

The limits the authors state

The authors list several. The method assumes the subtasks combine clearly, and it works best on composite tasks that split into parts. The tests used one blocksworld, and the authors say measuring on hundreds of tasks across tens of domains is future work. Thinking models were left out because they often made up the environment's replies, and the test cannot tell a habit from a deliberate choice. A model could also show full goal-directedness on one breakdown of a task and still not be fully goal-directed overall.

Two other views: a mechanistic probe and a philosophical objection

Xu and Rivera take a different route. They propose that a policy is goal-directed if it is well modelled as near-optimal for many sparse reward functions, test this in small Markov decision processes, and then fine-tune LLaMA-2-7B on two datasets with sparse versus dense losses. A classifier trained on the model's activations could separate the two kinds, with some noise when moved to other data. They call this preliminary evidence and say the most rigorous existing definitions are hard to compute in real settings.

Rajcic and Søgaard argue the whole project has a ceiling. Behavioral measures depend on how goals and agents are formalized, and mechanistic probes depend on what internal states mean. Using a mouse and a piece of cheese, they show how a goal can be described at many granularities: this cheese, any cheese, or just not being hungry. They also show cases where a measure gives odd results, such as a stone in place of the mouse. Their conclusion is that goal-directedness "cannot be measured objectively" and indicates only how well a formal model fits the system. They suggest studying how goal-directed behavior emerges in simulation.

Read these three as three answers to one question rather than a verdict. The first measures behavior against capability, the second looks inside, and the third doubts that either gives an observer-free number. Our posts on coherence theorems and agent foundations cover the older theoretical side, and mesa-optimization covers a related inner-alignment worry.

Frequently asked questions

Is a language model goal-directed?

Partly, and it depends on the model and the task. In the DeepMind study most models were not fully goal-directed, and the most goal-directed models still fell somewhat short on some tasks.

How is goal-directedness different from capability?

Capability is what a model can do. Goal-directedness is how fully it uses that ability on a given goal. The study measures it by comparing the full task with the same skills tested alone.

Does a motivating prompt make a model goal-directed?

Only somewhat. A prompt telling the model to try hard improved results in the study, but not to the level of full use of its capabilities.

Can goal-directedness be measured at all?

Researchers disagree. One paper offers a measure for LLMs, one proposes a probe of internal states, and a third argues no measure can be fully objective.

Get started

To build the background that makes papers like these easier to read, start with the short lessons and work up from the basics. You sign in with Google or an emailed code, and it is a web app. Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.