Joseph Weizenbaum: the ELIZA creator who questioned AI

Joseph Weizenbaum was the MIT computer scientist who wrote ELIZA, one of the first programs you could chat with, and then became one of artificial intelligence's leading critics. What turned him was the way people treated his simple program as if it understood them. This post covers his life, how ELIZA worked, what shocked him, and the argument he spent the rest of his career making: that some choices should never be handed to a computer, however capable it gets. Everything here comes from Wikipedia's articles on Joseph Weizenbaum, his book Computer Power and Human Reason, and the ELIZA effect.

If you want to feel the pull of those choices yourself, Contain ASI is a collectible card game in your browser where you run a frontier AI lab and race rival labs to superintelligence, and where cutting corners is the fastest way to win and the surest way to lose the future.

Who was Joseph Weizenbaum?

Weizenbaum was born in Berlin on 8 January 1923 to Jewish parents. In January 1936 he escaped Nazi Germany with his family and settled in the United States. He began studying mathematics in 1941 at Wayne State University in Detroit. In 1942 he broke off to serve in the U.S. Army Air Corps as a meteorologist. He had been turned down for code work because of his "enemy alien" status.

After the war he finished his degrees, a B.S. in 1948 and an M.S. in 1950. He worked on analog computers, helped build a digital one, and in 1956 worked for General Electric on ERMA, a system that let banks process checks automatically using magnetic ink. In 1962 he wrote a short paper with a telling title, "How to Make a Computer Appear Intelligent", about a program that played the board game Gomoku well enough to beat beginners.

In 1963 MIT hired him as an associate professor on the strength of SLIP, a list processing language he had written. By 1970 he was a full professor. He also taught at Harvard, Stanford and the University of Bremen. In 1996 he moved back to Berlin, near the neighborhood where he grew up, and he died in 2008 at 85.

ELIZA and the DOCTOR script

In 1966 Weizenbaum published ELIZA, written in his own SLIP language. He named it after Eliza Doolittle, the young woman in George Bernard Shaw's play Pygmalion who is taught to speak like someone she is not.

ELIZA did not understand anything. It looked for keywords in what you typed, then used two kinds of rule. A decomposition rule split your sentence around the keyword. A reassembly rule built a reply, often from your own words. Which keywords mattered, and how replies were built, came from a separate script.

The famous script was DOCTOR. It copied the style of the psychologist Carl Rogers, who used open questions to get patients talking. That style suits a program with no understanding, because it mostly hands your words back to you as a question. You can read more about the program itself in our post on how ELIZA fooled people.

What shocked him: the ELIZA effect

What came next shocked Weizenbaum. Many users took ELIZA seriously and opened their hearts to it. The best known story is about his secretary. She knew it was a simulation, yet while using it she asked him to leave the room.

He later wrote: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."

Others hailed ELIZA as an early step toward thinking machines. Weizenbaum thought that was a misreading, and much of his later writing tried to correct it. The habit he saw now has a name, the ELIZA effect: in one definition, "the susceptibility of people to read far more understanding than is warranted into strings of symbols" produced by computers.

His wider worries about computers

From there Weizenbaum began to think about what artificial intelligence, and computers in general, would do to society. In an interview with MIT's newspaper, The Tech, he went further than AI. He argued the computer was at heart a conservative force. His example was banking. Computers let banks handle an ever larger flood of checks, so they never had to reorganize, for instance by decentralizing. The system got more efficient but never had to change.

He also worried about the military's role, calling the computer "a child of the military". He objected to words like "defense" for organizations whose work includes killing people. Plain words, he believed, would make people slower to reach for violence. He said he was not a pacifist and accepted that arms are sometimes needed.

Computer Power and Human Reason: deciding vs choosing

His 1976 book, Computer Power and Human Reason: From Judgment to Calculation, set out his case. Its central move is to set aside the question most people argue about. Whether artificial intelligence is achievable, he said, is irrelevant to whether a computer should be given a particular task.

He drew a line between two things:

• Deciding is computational. In the end it can be programmed.
• Choosing is the product of judgement, not calculation. Defining a task and picking the criteria for doing it well is a creative act that rests on human values.

So when someone hands a decision to a computer, that act is itself a choice. It reflects the values of whoever made it, and it has real consequences for the people the computer then decides about. In the book's summary, AI may be possible, but computers should never be allowed to make important decisions, because they will always lack human qualities such as compassion and wisdom.

Try it: sort tasks into deciding and choosing

You can test his distinction in five minutes. Take a system that screens job applications and write down every step it involves:

1. Reading each application and pulling out the dates and qualifications.
2. Checking each one against a list of required qualifications.
3. Writing that list of required qualifications in the first place.
4. Deciding whether to let software reject applicants with no person looking.
5. Telling a rejected applicant why.

Now mark each step "deciding" or "choosing" by Weizenbaum's test: could it be written as a procedure, or does it rest on someone's values? Steps 1 and 2 are deciding. Step 3 is choosing, because someone has to judge what matters in a candidate. Step 4 is the choice his book is about: handing the decision over. Step 5 is open to debate, and arguing it out is the point of the exercise. Wherever you land, you will notice that every "deciding" step sits inside a frame that a person chose.

Where people disagree, and his legacy

The book caused disagreement with, and a split from, many in the AI research community. Weizenbaum later said he came to take pride in that. Early academic reviews said the book made readers see that even very capable computers lack the moral sense and empathy some judgments need. Readers still differ on where his line should fall, and on whether "deciding" and "choosing" can always be told apart.

His influence outlasted the argument. He won the 1988 Norbert Wiener Award for Professional and Social Responsibility from Computer Professionals for Social Responsibility. Two documentaries cover his life and ideas: Weizenbaum. Rebel at Work. (2007) and Plug & Pray (2010), about the ethics of AI. A German research institute on the networked society, the Weizenbaum Institute, carries his name. For other critics and pioneers of the same era, see our posts on Hubert Dreyfus's critique of AI and John McCarthy, and our overview of the main questions in AI ethics.

Frequently asked questions

What is Joseph Weizenbaum known for?

He wrote ELIZA, an early program people could chat with, published in 1966. He is also known for his 1976 book Computer Power and Human Reason and for his criticism of handing human judgment to computers.

Why did Weizenbaum turn against AI?

He was shocked that people, even those who knew ELIZA was a simulation, confided in it and took it seriously. That led him to argue that some decisions must stay with people, whatever computers can do.

What is the difference between deciding and choosing?

In Weizenbaum's terms, deciding is computational and can be programmed. Choosing is judgement based on human values, and putting a computer in charge of a decision is itself a choice.

What is Contain ASI?

It is a collectible card game in your browser where you run a frontier AI lab and race rival labs to superintelligence. You sign in with a Google account to play. It is a game, not a forecast.

Get started

Weizenbaum's question was who stays responsible as machines get more capable. Contain ASI puts a version of it on the table as a card game. You run a frontier AI lab and race rival labs to superintelligence. The game's own page puts it plainly: the race is the easy part, alignment is the hard part. Cutting corners is the fastest way to win, and the surest way to lose the future. Its Why this game page says you learn why capability races are hard to stop, which levers governance has, and why risky research is tempting. You play in your browser with a Google account, and your cards and Credits are the same on your phone and your computer.

Play Contain ASI: a collectible card game in your browser where you run a frontier AI lab and race rival labs to superintelligence.