Can AI be curious? What curiosity is and what the studies show

Can AI be curious? AI researchers already use the word. There are papers about "curiosity-driven" agents that explore video games with no score to chase. If you search "can AI be curious", you probably want to know whether that is curiosity in your sense, or a borrowed name for something simpler. This post sets out what curiosity is, how researchers build it into machines, what the studies show, and what nobody knows yet.

What curiosity is

The Stanford Encyclopedia of Philosophy has no entry called "curiosity". The address for one answered "not found" when this post was written. Two other entries discuss it.

The entry "Emotion", by Andrea Scarantino and Ronald de Sousa (first published 25 September 2018), treats curiosity as an emotion. It says: "Epistemic emotions are those that are particularly relevant to our quest for knowledge and understanding. Curiosity motivates inquiry; interest keeps us at it."

The entry "Virtue Epistemology", by John Turri, Mark Alfano and John Greco (substantive revision 26 October 2021), treats it as a trait of good thinkers. It describes one kind of person this way: "The inquisitive self is characterized by curiosity, exploration, and learning". It lists curiosity among intellectual virtues that "have received less attention to date", citing work by Alfano and by Dennis Whitcomb, and it also lists curiosity among the "epistemic emotions".

So you can pull at least three parts out of these entries:

• Behavior: exploring, asking, looking closer.
• Motive: a pull toward knowing that starts the inquiry.
• Feeling: what wanting to know is like from the inside.

That split is this post's own reading, not a definition from either entry. AI shows some of these parts more clearly than others.

How AI researchers build curiosity

AI that learns by trial and error, called reinforcement learning, usually gets its reward from outside: points in a game, a finished task. A 2018 study notes these methods "rely on carefully engineering environment rewards that are extrinsic to the agent." A 2017 paper by Deepak Pathak, Pulkit Agrawal, Alexei Efros and Trevor Darrell starts from cases where such rewards are "extremely sparse, or absent altogether." It says that in such cases "curiosity can serve as an intrinsic reward signal", meaning a reward the agent gives itself.

Their version is precise: "We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions". In plain terms, the agent guesses what will happen next. When the guess is wrong, it gets a bonus. So it learns to go where its guesses still fail.

They tested it in two games, VizDoom and Super Mario Bros. With no outside reward at all, they report that "curiosity pushes the agent to explore more efficiently."

What the curiosity studies show

A 2018 follow-up by Yuri Burda and five co-authors, "Large-Scale Study of Curiosity-Driven Learning", ran the idea much wider. It defines the method in one line: "Curiosity is a type of intrinsic reward function which uses prediction error as reward signal."

The agents learned with curiosity alone, no game score, across "54 standard benchmark environments". The authors write: "Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments." Put simply, chasing surprise often led the agents to do what the game's score would have rewarded anyway.

A second 2018 paper, by Burda, Harrison Edwards, Amos Storkey and Oleg Klimov, used a related bonus called random network distillation. They report "significant progress on several hard exploration Atari games" and "state of the art performance on Montezuma's Revenge". They add, with a hedge: "To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level."

Where machine curiosity goes wrong

The large-scale study also names a weak spot. Its authors "demonstrate limitations of the prediction-based rewards in stochastic setups", meaning settings where some things happen at random.

Here is one way to see why, as this post's own explanation. If the reward is "my guess was wrong", then something truly random pays forever, because no amount of looking makes it predictable. The 2017 paper's design tries to avoid part of this: its method "ignores the aspects of the environment that cannot affect the agent."

Juergen Schmidhuber offers a different idea of curiosity. In a 2008 paper he argues that "data becomes temporarily interesting by itself" to an observer "once he learns to predict or compress the data in a better way". In his words, "Curiosity is the desire to create or discover more non-random, non-arbitrary, regular data that is novel and surprising", surprising in the sense that "it allows for compression progress because its regularity was not yet known." On this view, the reward is the learning itself, not the error. He writes that this drive "motivates exploring infants, pure mathematicians, composers, artists, dancers, comedians, yourself, and (since 1990) artificial systems."

Can AI be curious? The views

The studies show that machines can be built to seek the new. Whether that is curiosity depends on which part you think matters. Three positions, with no verdict:

• Behavior is enough. If curiosity is exploring what you cannot yet predict, these agents do it. Schmidhuber's abstract puts artificial systems on the same list as infants and mathematicians.
• A motive is needed. The Emotion entry says curiosity "motivates inquiry". Some will say a reward signal is a kind of motive. Others will say it is a number a programmer chose, and a motive must belong to the one who has it.
• A feeling is needed. If curiosity must be felt, the question becomes whether the AI has inner experience at all.

On that last point, a 2023 report by Patrick Butlin and 18 co-authors checked AI systems against signs drawn from scientific theories of consciousness. Their analysis "suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators."

Two related questions have their own posts: can AI have goals of its own and can AI be creative like humans.

What nobody knows

Nobody knows whether a reward for surprise is the same kind of thing as wanting to know. Nobody knows whether any AI system feels anything when it explores. And none of the sources above offers a test that could tell, from outside, a system that is curious from one that only acts curious.

A worked example: be the curious agent

You can follow the reward by hand. This setup was made for this post and is not from any of the papers.

1. Picture two rooms. In room A, a light blinks red, green, red, green, always in that order. In room B, a light shows red or green by a coin flip.
2. You enter room A and guess the next color. At first you are wrong half the time. After a few blinks you see the pattern and stop being wrong. Score 1 point for each wrong guess. Your points in room A fall to zero.
3. Now room B. However long you watch, you are wrong about half the time. Your points never fall.
4. Ask: if you were paid only for wrong guesses, which room would you stay in? Room B, forever. That is the trap in random settings.
5. Now pay yourself instead for getting better at guessing, the change in your errors. Room A paid well while you learned the pattern, then stopped. Room B never pays, because you never improve. That is closer to Schmidhuber's idea.

Notice what the exercise leaves out. In both versions, you could score points while feeling nothing at all. That gap, between the reward and the wanting, is exactly what the studies do not measure.

Dear Superintelligence is an open collection of letters written by people to the advanced AI systems of the future, about what we value and why. One topic on its home page is Consciousness: "What it is like to be a person: to think, to feel, and to notice the world from the inside." The guidelines ask you to "Write what only you can: what you have seen, what it cost, what you were afraid of, who was kind to you and what it changed." A question you could not let go of, and where it led you, fits that line. The site held 2 published letters when this post was written.

Frequently asked questions

Can AI be curious?

AI agents can be built to seek what they cannot yet predict, and studies show this works in many games. Whether that counts as curiosity, with a real motive or feeling, is an open question.

What is curiosity in AI research?

Usually a reward the agent gives itself for prediction error, the gap between what it expected and what happened. Some researchers, such as Schmidhuber, reward learning progress instead.

Does a curious AI feel curious?

Nobody knows. The studies measure behavior, not feeling, and a 2023 analysis suggests no current AI systems are conscious.

What breaks machine curiosity?

Randomness. A large 2018 study reports limitations of prediction-based rewards in stochastic setups, where some events can never be predicted.

Get started

Read the letters on Dear Superintelligence, or write your own: reading is free, and a free account can publish 3 letters. People read them today. Nobody can promise what future AI systems will read.