Turing test: what it is and why people argue about it

The Turing test is one of the most quoted ideas in computing, and one of the most argued over. It asks whether a judge reading a typed conversation can tell a machine from a person. This guide explains how the test works, where it came from, why researchers still disagree about it, and how to read a headline that says a machine "passed" it.

What is the Turing test?

Wikipedia's article on the Turing test describes it as a way to assess whether a machine can show intelligent behaviour equal to a human's, by imitating a conversation. In the modern version, a human judge reads a text transcript of a natural-language conversation between a person and a machine. The judge tries to pick out the machine. The machine passes if the judge cannot reliably tell the two apart.

One detail matters a lot: the result does not depend on whether the machine answers correctly. It depends only on how closely its answers resemble a human's. A wrong answer a person might give counts for the machine, not against it.

Where the Turing test came from

Alan Turing introduced it as the "imitation game" in his 1950 paper "Computing Machinery and Intelligence", written while he worked at the University of Manchester. (For more on the man, see Alan Turing's contributions to computer science.) It opens with "I propose to consider the question, 'Can machines think?'"

Turing then sets the question aside. Because "thinking" is hard to define, he replaces it with one "closely related to it and expressed in relatively unambiguous words". He borrows a party game in which an interrogator questions a man and a woman in another room to work out which is which. His new question is whether imaginable digital computers could do well in that game. The rest of the paper answers the major objections to the idea that machines can think; the article counts nine.

Older roots

The article traces the idea back much further. René Descartes wrote in 1637 that a machine could utter words, but would never arrange its speech to reply appropriately to everything said in its presence, as even the dullest person can. Denis Diderot wrote in 1746 that a parrot able to answer everything would be, without hesitation, an intelligent being. In Jonathan Swift's 1726 novel Gulliver's Travels, a king suspects Gulliver is a clockwork machine, and relaxes only after several questions get rational answers.

Turing's other versions

Turing gave no single fixed form. In a 1952 BBC radio broadcast he described a jury that questions a computer, where the computer must make a significant share of the jury believe it is a man.

A worked example: judge a conversation as Turing imagined it

Turing imagined his own sample of the questioning. The article quotes it:

Interrogator: In the first line of your sonnet which reads, "Shall I compare thee to a summer's day," would not "a spring day" do as well or better? Witness: It wouldn't scan. Interrogator: How about "a winter's day". That would scan all right. Witness: Yes, but nobody wants to be compared to a winter's day.

Notice what the questions test. They are about sound, taste and feeling, not arithmetic. The article points out that Turing chose poetry, not mathematics, as his example. Now try the test yourself on any chat transcript, using these six questions:

1. Did the judge know a machine might be present? Turing never says whether the judge is told, and the article notes this changes the outcome.
2. How good was the judge? A questioner who does not suspect anything is easy to fool.
3. How long did it run? The contest rounds the article describes ran from five minutes to more than twenty.
4. Could the machine hide behind tricks? Dodging, changing the subject or copying typing errors can look human without any understanding.
5. Were real people mistaken for machines? This happens, and the article calls it the confederate effect.
6. What did "passing" mean? Was it fooling one judge once, or a fair share of judges in repeated trials?

A claim that passes all six is much stronger than one that passes two.

Why people argue about the Turing test

Turing did not say outright that his test measures intelligence; he wanted a clear alternative to the word "think" to answer critics. Yet it has been used as a measure of intelligence, and philosophers and computer scientists have criticised it. The main objections are below.

The judge can be fooled

The result can be dominated by the attitudes, skill or naivety of the questioner. The cognitive scientist Gary Marcus is quoted as saying it only shows how easy it is to fool humans. Simple chat programs such as ELIZA have repeatedly convinced people who did not know they were talking to a computer.

It tests imitation, not intelligence

The test checks whether a machine behaves like a human, and human behaviour is not all intelligent. A machine must copy mistakes, insults and typing errors to pass. And some intelligent behaviour is inhuman: a machine that solved a problem no person could solve would be spotted at once and fail.

The Chinese room

In 1980 the philosopher John Searle argued, with his "Chinese room" thought experiment, that the test cannot show a machine thinks. A program could pass by manipulating symbols it does not understand. The article notes that his argument, like the test itself, has been both widely criticised and backed. Turing saw this coming: he wrote that there is "something of a paradox" about consciousness, but that it need not be solved to answer his question.

Language only

The test covers only linguistic behaviour. Other abilities are not tested at all.

A distraction for researchers

Stuart Russell and Peter Norvig write that AI researchers have devoted little attention to passing the test. Their comparison: planes are tested by how well they fly, not by whether they can fool pigeons. John McCarthy added that the philosophy of AI is unlikely to affect AI research any more than the philosophy of science affects science.

A real contest showed several of these problems at once. The Loebner Prize ran from 1991, and its first winner was a program with no identifiable intelligence that won partly by imitating human typing errors. Its final contest was in 2019, after funding ran out following Hugh Loebner's death in 2016.

Variations and alternatives

People have proposed many other tests, each fixing one of the problems above:

• Reverse Turing test. The roles swap. The CAPTCHA, the distorted letters a website asks you to type, is a form of this: it tries to tell whether the one typing is a person.
• The Feigenbaum test. The machine's answers must be indistinguishable from an expert's in one field, such as literature or chemistry.
• The Total Turing test. Stevan Harnad adds perception and the ability to handle objects, so the machine needs vision and robotics too.
• The minimum intelligent signal test. Only yes or no answers are allowed, which avoids chat tricks, though it works more like an IQ test than a conversation.

Narrower tests also help when you ask how general a system really is. For that distinction, see narrow AI vs general AI and what AGI is.

What the Turing test means for AI safety

The test is about talk, but it points at a safety question. Because it asks a machine to seem human, it cannot measure a machine that is smarter than people. The article is blunt: a machine more intelligent than a human would have to deliberately avoid appearing too intelligent to pass, so the test cannot be used to build or evaluate such systems. Several alternatives for superintelligent systems have been proposed.

A second worry is about what a measure rewards. Erik Brynjolfsson calls it the "Turing Trap": by focusing on imitating humans instead of extending what they can do, the test risks steering research toward technology that replaces workers, which could lower their wages and their political power.

The article also records a hopeful reading. Turing's sonnet example shows his interest in empathy and aesthetic sensitivity, and some have suggested these may matter for a "friendly AI". If that idea is new, see friendly AI: what the term means.

Stories about machines passing as people are much older than computers, and fiction has shaped how people think about them. One way to explore the risk side is as a story: Contain ASI is a story strategy game for 1 to 4 players that runs in your browser. Its home page sets the scene: "January 2024. Four AI labs. One race. For three years, keep every lab from crossing into recursive self-improvement before 2027."

The game is fiction. Its footer says it is inspired by the AI 2027 scenario and that all of its labs, people and events are made up.

Frequently asked questions

What is the Turing test in one sentence?

A judge reads a typed conversation between a person and a machine, and the machine passes if the judge cannot reliably tell which is which.

Who invented the Turing test?

Alan Turing introduced it as the "imitation game" in his 1950 paper "Computing Machinery and Intelligence".

Does passing the Turing test mean a machine thinks?

Not necessarily. Turing himself did not say it measures intelligence, and Searle's Chinese room argues that producing the right replies is not the same as understanding.

Is Contain ASI a Turing test?

No. It is a story strategy game, and its page says all of its labs, people and events are fictional.

Get started

If you would rather explore the AI race as a story than as an essay, Play Contain ASI: a story strategy game for 1 to 4 players that runs in your browser. Its labs, people and events are all fictional. You will need a recent Chrome, Edge, Firefox or Safari, because it runs on WebGL 2.