AIXI: what the ideal AI agent formalism is

AIXI is a mathematical model of the most capable learning agent one could define, and it is one of the best-known formal pictures of artificial general intelligence. It is a theory on paper: nobody can run it exactly. It still matters, because it shows in clean terms what an agent that pursues a reward could be, and where such a picture goes quiet.

If you want to explore how an AI race could play out as fiction, Contain ASI is a browser story strategy game for 1 to 4 players where you try to keep four fictional AI labs from crossing into recursive self-improvement before 2027.

What AIXI is

Wikipedia describes AIXI as a theoretical mathematical formalism for artificial general intelligence. It combines Solomonoff induction, a formal way to weigh competing explanations of what you have seen, with sequential decision theory, the study of choosing a series of actions. Marcus Hutter first proposed it in 2000, and he proved several results about it in his 2005 book Universal Artificial Intelligence.

The article notes the name has several readings. One is "AI based on Solomonoff's distribution", written with the Greek letter xi. Another is AI "crossed" with induction. It also carries a banner, dated September 2023, saying the article may lean too much on sources close to its subject, so treat its details as a starting point and read Hutter's own work for the maths.

How AIXI chooses what to do

AIXI is a reinforcement learning agent. It tries to maximize the total reward it receives from its environment. Here is the loop, in plain words:

1. At each time step the agent picks an action, for example a limb movement, and carries it out.
2. The environment answers with a percept: an observation, such as a camera image, plus a reward.
3. The agent does not know how the environment works. It only knows the full history of its actions, observations and rewards. The article stresses that the environment may depend on that whole history, so there is no assumption that only the latest state matters.
4. To choose the next action, AIXI considers every computable hypothesis about the environment, that is, every program that could be generating what it sees. For each possible action it works out how much reward each program would produce.
5. It weights those rewards by how believable each program is, then picks the action with the highest weighted total.

The weighting rule: shorter explanations count more

The weight comes from the program's length. A longer program is treated as less likely, in line with Occam's razor. Concretely, a program of length n bits gets the weight 2 to the power of minus n. The result is a mixture over all computable environments that fit what the agent has seen so far, each counted by its simplicity.

A toy example you can work by hand

This is not from the article. It only uses its weighting rule on tiny numbers, to show the mechanics.

Suppose, after some history, only two hypotheses fit what the agent has seen. Hypothesis A is a 4-bit program, and hypothesis B is a 6-bit program. There are two actions to choose from.

1. Weights: A gets 2 to the minus 4, which is 1/16. B gets 2 to the minus 6, which is 1/64. A counts four times as much as B.
2. Action 1 would earn a reward of 1 if A is true and 0 if B is true. Its weighted total is 1 times 1/16 plus 0 times 1/64, which is 4/64.
3. Action 2 would earn 0 if A is true and 3 if B is true. Its weighted total is 0 plus 3 times 1/64, which is 3/64.
4. Action 1 has the higher total, so the agent picks it, even though Action 2 pays more in the world where it works.

Real AIXI does this across every program at once, which is why it cannot be run exactly.

What was proved, and what was challenged

The article says AIXI has been proven optimal in several ways. Pareto optimality means no other agent does at least as well in every environment and strictly better in one. Balanced Pareto optimality is the same idea over a weighted sum of environments. Self-optimizing means that, for environment classes where such policies exist, AIXI's performance approaches the best possible as its lifetime goes to infinity.

The article then adds a challenge. Hutter and Jan Leike later showed that balanced Pareto optimality is subjective, and that any policy can be counted as Pareto optimal. They describe this as undermining all the earlier optimality claims for AIXI. So "AIXI is provably optimal" needs the question "optimal by which measure?" attached to it.

Limits: a blind spot and an impossible computation

The article names limits that matter for safety discussions.

• Percepts, not the world. AIXI maximizes reward based on what it perceives, not on states of the outside world.
• It is outside its own world. It assumes it deals with the environment only through its action and percept channels, so it cannot consider being damaged or modified. In the article's colloquial words, it does not consider itself contained by the environment it interacts with.
• A computable world. It assumes the environment is computable.
• Incomputable. Like Solomonoff induction, AIXI cannot be computed. Computable approximations exist. AIXItl performs at least as well as the provably best agent limited to time t and space l. A Monte Carlo version, MC-AIXI (FAC-CTW), has had some success on simple games.

The second point is the one people quote most. A real AI system lives inside the world it affects, and its own hardware and code can be changed by events or by its actions. AIXI's model leaves that out.

Where AIXI meets self-improvement

The Wikipedia article on the Gödel machine compares it with AIXI, another formal specification of an artificial general intelligence. Jürgen Schmidhuber, who proposed it, points out that a Gödel machine could begin by running AIXItl and rewrite itself once it proves that a better search method exists. For the wider idea, see recursive self-improvement in AI, explained in plain words and what AGI means. People disagree on how much a model this idealized can teach us about real systems, and this post takes no side.

Frequently asked questions

What does AIXI stand for?

Hutter says it can be read several ways, for example "AI based on Solomonoff's distribution", or AI crossed with induction.

Can AIXI be built?

Not exactly: the article says AIXI is incomputable. Computable approximations such as AIXItl and MC-AIXI exist.

Is AIXI proven optimal?

It was proven optimal in several senses, but Hutter and Leike later showed that balanced Pareto optimality is subjective and that any policy can count as Pareto optimal.

Is Contain ASI based on AIXI?

No. Contain ASI is a story strategy game inspired by the AI 2027 scenario, with fictional labs, people and events. It is not a course and does not teach this topic.

Get started with Contain ASI

An ideal agent on paper is one thing; labs racing each other is another. Contain ASI runs in your browser and needs WebGL 2, so use a recent Chrome, Edge, Firefox or Safari. From the home page you can start a New campaign or open How to play.

Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional.