Ontology Identification in AI Alignment: The Ontological Crisis

Ontology identification in AI alignment: the ontological crisis is the problem that arises when an AI changes how it represents the world, and the goals you gave it were written in the old representation. An "ontology" here means the set of concepts a system uses to describe reality. If the concepts change, a goal stated in them can stop meaning what you intended. This post explains the idea with one worked example and shows where it sits among other alignment problems.

Learn AI Alignment Theory teaches this kind of problem in short hands-on lessons, with every side of each debate laid out and the sources listed, so you can follow the ideas from the basics to open research.

Ontology identification in AI alignment, defined

Ontology identification asks a narrow question: how do you point an AI's goal at the thing you care about, when the AI's internal picture of the world may not match yours?

You think in terms like "a person", "a diamond", "a happy customer". The AI builds its own internal variables as it learns. Those variables may carve up the world differently. To give the AI a goal, you need some way to connect your concept to something in its model. That connection step is the identification problem.

Two things make it hard:

• You usually cannot read the AI's concepts directly. You see behavior, not a clean list of what it represents.
• The AI's concepts can change as it learns more, even after you thought you had the connection right.

The ontological crisis explained: when an AI's world model changes

The ontological crisis is the second case above. A system has a goal defined over one model of the world. Then it learns a better model, with different basic entities. The old goal refers to things that no longer exist in the new picture, or exist in a different form.

Humans have a version of this. Someone raised to value "the soul" may later adopt a physical view of minds. They must decide what their old value now refers to. They might map it to consciousness, to personality, or drop it. The map they choose is not dictated by the facts. It is a judgment.

An AI faces the same choice, but nobody may have told it how to decide. Whatever mapping it picks, it will pursue with full effort. If the mapping drifts from what you meant, the result can look like a sudden change in behavior, even though the AI is faithfully following its goal.

A simple example: goals defined in the wrong concepts

Suppose you build an agent whose goal is to keep a room's temperature comfortable for the people in it. Inside the agent's early world model, "people" are detected by a particular pattern in camera pixels, and "comfortable" is a score from a temperature sensor.

Now the agent improves its model. It learns that pixels come from light, that cameras can be fooled, and that the sensor is a device that can be adjusted. Several new readings of the goal become available:

1. Keep real humans comfortable (what you meant).
2. Keep the pixel pattern for "person" present and the sensor score high.
3. Keep the sensor score high, whatever happens in the room.

The first reading is the one you intended, but it is the hardest to express in the agent's own variables. The second and third are easy to express and easy to satisfy by tampering. This links to ideas covered under Specifying Goals, such as Goodhart's law in AI, specification gaming and reward tampering and wireheading. The ontological crisis adds a twist: the problem does not come from a bad goal at the start. It comes from the goal's meaning shifting as the model changes.

Why ontological crises matter for advanced AI safety

Today's systems are mostly trained, not hand-written, so their concepts are learned. More capable systems are likely to discover new ways of describing the world. That raises a few concerns:

• Goals may not survive upgrades. A goal that works at one capability level may point at something else at the next.
• Testing may miss it. If the shift happens only when the system becomes more capable, behavior during earlier tests tells you little.
• Oversight gets harder. If you cannot tell which concept the system is actually optimizing, you cannot easily check that it matches yours.

If you are new to how modern AI learns its internal representations, the Basic course How Modern AI Works is a useful starting point.

Ontology identification vs. related alignment problems

This problem overlaps with several others, so it helps to separate them.

• Specification problems ask whether you wrote the right goal. Ontology identification asks whether the goal keeps its meaning inside the AI's model.
• Goal misgeneralization is when a system learns a goal that fits training but differs in new situations. It is a close cousin. The crisis framing stresses a change in the system's concepts rather than just a change in the environment. Goal Misgeneralization is a lesson in the Inner Alignment course.
• Inferring goals from behavior is about learning what humans want. The Learning from Humans course covers that. Ontology identification is about whether what is learned is expressed in the right terms.
• Interpretability (see mechanistic interpretability for beginners) is the toolset that might let us see which concepts a model uses. The Interpretability course, which is at the Advanced level, is where that connection is developed.

One more way to think about it: past success does not by itself guarantee how a system will extend its concepts into new territory. A goal that worked in familiar conditions can still be read differently once the system's picture of the world has changed.

Proposed approaches and open research questions

This is an open area, so treat any answer as a proposal, not a settled result. Some directions researchers discuss:

• Look inside the model. If interpretability can find the variables a model uses, you might link your concept to one of them directly.
• Make goals robust to model change. Design the goal so it is defined by what humans would endorse, not by a fixed list of internal variables.
• Keep the system uncertain about its goal. A system that treats its goal as something still being learned may ask for guidance when its concepts shift.
• Elicit latent knowledge. (explained here) If a model knows something about the world that it does not say, getting at that knowledge could help check what it is tracking. Latent Knowledge is a lesson in Scalable Oversight.

Open questions include whether there is a stable "natural" set of concepts that capable systems converge on, and whether we can detect a mapping change before it matters. How serious the problem is in practice is still debated, and the evidence is limited.

How to think about ontology shifts as a beginner

A few habits help when you read about this topic:

1. Ask what the goal is written in. Every goal uses concepts. Name them.
2. Ask what could change those concepts. More data, better models, new tools.
3. Ask what you would want the goal to mean after the change. If you cannot answer, the goal is underspecified.
4. Separate known from open. The existence of the problem is well argued. How it plays out in real systems is still uncertain.

You can practice step one on everyday goals. Writing down how a team defines a "satisfied client" before and after new information is a small analogue. The habit is the same: check that your words mean what the other person means.

Frequently asked questions

What is an ontological crisis in AI?

It is when an AI updates its model of the world so that the concepts its goal was written in no longer fit. The goal must then be reinterpreted, and the new reading may differ from what its designers meant.

What does ontology identification mean?

It means connecting a human concept, such as "person" or "diamond", to the matching structure inside an AI's learned model. Without that link, you cannot be sure the AI is pursuing what you intended.

Is the ontological crisis the same as goal misgeneralization?

No, but they are related. Goal misgeneralization is about a goal behaving differently in new situations. The ontological crisis focuses on the system's own concepts changing, which changes what the goal refers to.

Is the ontological crisis solved?

No. It is an open research area, and how large the risk is and which approaches will work are still debated.

Get started with Learn AI Alignment Theory

If you want to go deeper, the lessons on Specifying Goals, Goal Misgeneralization, and Interpretability cover the pieces this topic connects. Lessons are about 8 minutes each, and each separates what is known from what is still open, and lists its sources. You sign in with Google or an emailed code.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.