Chinese room argument: can a program understand?

The Chinese room argument is a famous claim in the philosophy of mind: a computer that runs a program cannot have a mind or understand anything, however human its answers look. It comes from the philosopher John Searle, who set it out in a 1980 paper called "Minds, Brains, and Programs". This post explains the thought experiment, the case Searle built on it, the main replies, and why most AI researchers do not treat it as a limit on what machines can do.

Every claim about the argument below comes from Wikipedia's article on the Chinese room, opened for this post. Where thinkers disagree, the post says so and gives no verdict.

What the Chinese room argument says

The argument says that running the right program is not enough to have a mind, understanding or consciousness. A computer could behave in a way that looks fully intelligent and still understand nothing.

Searle did not invent the worry from nothing. Wikipedia lists earlier versions: Gottfried Leibniz made a similar point in 1713 with a brain enlarged to the size of a mill, and the Soviet writer Anatoly Dneprov told a story in 1961 in which a stadium of people act as the switches of a translation program in a language none of them know.

The room, step by step

Start with a computer program that takes Chinese characters in and sends Chinese characters out. It does this so well that no one can tell it from a person who speaks Chinese. That means it would pass the Turing test, the old idea that a machine counts as thinking if it can hold a conversation without being found out. If you want that background first, read the Turing test and why people argue about it.

Now Searle puts himself in a closed room. He has an English copy of the program, plus pencils, paper, erasers and filing cabinets. Chinese characters slip under the door. He follows the program step by step, and it eventually tells him which Chinese characters to slide back out.

From outside, the room answers like a Chinese speaker. Inside, Searle does not understand a word of Chinese. He sees only shapes and rules. He argues there is no essential difference between his role and the computer's, since each simply follows a program. If he does not understand the conversation, he says, the computer does not either.

Searle's argument in four lines

The room is one step in a longer argument. In a version he gave in 1990, Searle starts with three plain statements and draws a conclusion.

• Programs are formal. They move symbols around by their shape, which is syntax, and pay no attention to what the symbols mean.
• Minds have meaning. Our thoughts are about things, which is semantics.
• Syntax by itself is neither the same as nor enough for semantics. The Chinese room is meant to prove this one: the room has syntax and, he says, no one in it knows what the symbols mean.
• So programs are neither the same as nor enough for minds.

The target is what Searle calls the strong AI hypothesis: that an appropriately programmed computer with the right inputs and outputs would have a mind in exactly the sense that people do. Futurists use "strong AI" differently, for machine intelligence that matches or beats people.

Searle does not say that only brains can think. He holds that brains cause minds through specific physical machinery, and he writes that "we are precisely such machines". His claim is that a formal program alone cannot do it.

The main replies

Wikipedia sorts the replies by what they try to show. Four of the best known are below, each with Searle's answer.

The system reply

The man does not understand Chinese, but the whole system of man, rulebook, paper and cabinets might. Searle answers by letting the man memorize the rules and do everything in his head. Then the "system" is just the man, and he still understands nothing.

The virtual mind reply

Marvin Minsky suggested a version: a mind can exist in a computer the way a virtual machine does, because software makes it appear there. Searle answers that a simulated mind is only a simulation, as a simulated rainstorm does not leave anyone wet. Minsky's own career is in Marvin Minsky: who he was and what he said about AI.

The robot reply

Put the program in a robot that moves around and senses the world, so its symbols connect to real things. Hans Moravec argued that meaning would then come from the physical world. Searle imagines that some inputs come from the robot's camera without the man knowing. He is still just following rules, and as Searle puts it, "he doesn't see what comes into the robot's eyes."

The brain simulator reply

Suppose the program copies, in fine detail, every neuron of a Chinese speaker's brain. Searle answers with a version using water pipes and valves that the man turns by rule, one valve per synapse. Where is the understanding, he asks, in the man or the pipes?

Doubts about the intuition

Some critics say the argument rests on a gut feeling. Ned Block writes that it depends on intuitions that certain things do not think. Daniel Dennett calls the room an "intuition pump" that works by picturing a case that is too simple. Others note that a man with pencils might need millions of years to answer one question, which makes the picture less obvious than it looks.

The other minds reply goes further. We never see another person's inner experience. We judge it from behavior. Turing called this the "polite convention that everyone thinks" and asked that machines get the same treatment. Searle replies that his opponents rely on intuitions too. Stevan Harnad, editor of the journal that ran the paper, observed that the overwhelming majority still think the argument is dead wrong.

Why most AI researchers set it aside

The argument is mainly about philosophy of mind, and Wikipedia says computer scientists and AI researchers generally assume it is not an issue for their field. The reason is simple: it does not show a limit on how intelligently a machine can behave. Stuart Russell and Peter Norvig wrote in 2021 that they are interested in programs that behave intelligently, and that making a machine conscious in exactly the way humans are is not a project they are equipped to take on.

The argument also leaves Newell and Simon's physical symbol system hypothesis untouched, because that is about outward intelligent action, not understanding. And today's deep learning programs do math on huge arrays of numbers and look little like the symbol processing of 1980. Nils Nilsson calls them "dynamic" rather than "symbolic", and some replies argue that Searle pictured a different kind of machine.

For safety questions, this matters in one direction. The argument is a claim about minds and understanding, not about power. Whether a system understands in Searle's sense and whether it can pursue goals well are separate questions. On the first, see Can AI be conscious?.

A thought exercise you can run

You can feel the argument with a small experiment, using paper only.

1. Write a rulebook of ten lines in a made-up code, such as "if you see the shape ▲●, write ■▲". Do not tell yourself what any shape means.
2. Ask a friend to slide you shape strings and answer by the book alone.
3. Now ask yourself: do you understand the exchange, or only the rulebook?
4. Next, tell yourself the shapes stand for rain and sun. Does anything change in what you do, or only in what you think you are doing?
5. Last, try the replies. Is the system of you, the book and your friend the thing that understands?

Whatever you decide, you have reproduced the real disagreement: whether following rules by shape can ever add up to meaning, or only to behavior that looks like it.

Frequently asked questions

Who came up with the Chinese room argument?

John Searle published it in 1980 in the paper "Minds, Brains, and Programs". Earlier thinkers, including Leibniz and Anatoly Dneprov, made similar arguments.

What is the main point of the Chinese room?

That following a program, which only manipulates symbols by their shape, is not enough for understanding or a mind. Searle says the person in the room understands no Chinese, and a computer running the same program would not either.

Does the Chinese room show that machines cannot think?

Not in general. Searle says machines could have minds if they reproduced what brains do, since "we are precisely such machines". His claim is about a program alone, and many thinkers reject even that.

Why do many AI researchers ignore the argument?

Because it does not limit how intelligent a machine's behavior can be, and their goal is useful, intelligent programs, not conscious ones.

Get started

The Chinese room asks whether a convincing machine really understands. A different question is what happens if the race to build such machines goes badly. You can explore that second question as fiction.

Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. Chapter 1, Before ASI, starts in January 2024 with four AI labs in one race, and you try to keep every lab from crossing into recursive self-improvement before 2027. It needs a browser with WebGL 2, such as a recent Chrome, Edge, Firefox or Safari.

It is a game, not a forecast, and it is not a course. For the philosophy itself, go back to the source: Wikipedia's article on the Chinese room.