What Is Chain of Thought Prompting? A Plain Guide for Founders

What is chain of thought prompting? It is asking a language model to write out its middle steps before it gives an answer, either by showing it worked examples or by adding a line such as "think step by step". On many multi-step problems, like arithmetic or logic puzzles, that small change makes answers more accurate. It also gives you steps you can read, though those steps are not always the real reason behind the answer.

The How Modern AI Actually Works course on Age of AI Entrepreneur explains next-token prediction, the idea that makes chain of thought work, in short lessons with their sources; sign in with your email and an emailed code to open it. This guide covers what the technique is, why it helps, where it misleads, and how to try it in your own AI product.

What is chain of thought prompting, in one example

The term comes from a January 2022 paper, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. The authors define a chain of thought as a series of intermediate reasoning steps. Their method was simple: in the prompt, they included a few example questions, each followed by worked steps and then the answer. The model copied the pattern on a new question.

Here is the difference in practice. A direct prompt asks a question and wants the answer only. A chain of thought prompt asks the same question but lets the model write the working first.

The question is the same in both. The only change is that the second prompt makes room for the steps, and the steps are now on the page for you to check.

Why writing the steps helps a model

A language model writes one token at a time, and each new token is predicted from all the text before it. That is the core of next-token prediction. When a model must jump straight to a final number, it has to get there in one step of prediction. When it writes "30 x 22 = 660" first, that line becomes part of the text it reads before predicting the answer.

So each written step works a bit like scratch paper. The model does not have a hidden notepad by default. The text it writes is its notepad. This is also why the technique fits the same machinery as the debate over whether ChatGPT is just autocomplete: the same next-word engine behaves very differently depending on what text it has to build on.

The 2022 paper reported that chain of thought prompting improved three large models on arithmetic, commonsense and symbolic reasoning tasks. Its largest model, with 540 billion parameters, reached state of the art accuracy on GSM8K, a set of math word problems, from just eight worked examples in the prompt. The authors also reported that the benefit appears in sufficiently large models, not small ones.

Few-shot and zero-shot chain of thought

There are two common ways to ask for a chain of thought.

• Few-shot: you paste a handful of solved examples, each with its steps, before the new question. This is the original 2022 method. It takes more effort but shows the model exactly the format you want.
• Zero-shot: you give no examples and add one instruction. A May 2022 paper, Large Language Models are Zero-Shot Reasoners, added "Let's think step by step" before each answer. With one large model, accuracy on the MultiArith set rose from 17.7 percent to 78.7 percent, and on GSM8K from 10.4 percent to 40.7 percent.

Those numbers come from models of 2022 on specific test sets. Newer models are trained in different ways, and some already work through steps without being asked. Treat the results as proof the idea can matter a lot, not as a promise for your model.

Where the written steps can mislead

It is tempting to read a chain of thought as the model's real reasoning. Research says to be careful, and researchers do not all draw the same line.

A 2023 paper, Language Models Don't Always Say What They Think, nudged models toward wrong answers, for example by reordering multiple-choice options so the correct one in the examples was always "(A)". The models followed the nudge but did not mention it in their steps. Instead they wrote plausible steps that justified the wrong answer. Accuracy dropped by as much as 36 percent on a suite of 13 tasks.

Another 2023 paper, Measuring Faithfulness in Chain-of-Thought Reasoning, changed the written steps, for example by adding mistakes, and watched whether the answer changed. Models varied a lot by task: sometimes they relied heavily on their steps, sometimes they mostly ignored them. The authors also found larger, more capable models gave less faithful reasoning on most tasks they studied, and concluded that chain of thought can be faithful when the model and task are chosen carefully.

So there are two fair readings. One: the steps are useful scratch work that often improves answers and helps you spot errors. The other: the steps are an explanation the model produces, which can look right while hiding what actually drove the answer. Both can be true at once, and your product should not depend on either one blindly. This sits next to the lesson on what AI gets wrong: a confident, well-written answer is not the same as a checked one.

A worked example: test chain of thought in your product

Say your tool reads a customer's usage and works out their monthly cost. Here is a one-afternoon test.

1. Collect 20 real questions with answers you have worked out by hand. Include a few with three or more steps.
2. Write two prompts. Prompt A asks for the number only. Prompt B adds: "Work it out step by step, then give the answer on the last line, after the word Answer."
3. Run both on all 20 questions, with the same model and the same settings. If you have changed the temperature setting, keep it fixed for both.
4. Score only the final answer, pulled from the last line, against your hand-worked answers.
5. Read five of Prompt B's wrong answers. Note where the steps went wrong, and whether any right answer came after a wrong step.
6. Weigh the cost. Steps add output tokens, so each request takes longer and costs more. Count the extra tokens per request.

If Prompt B is more accurate and the extra cost fits your margins, keep it. If the scores are the same, the shorter prompt wins. Either way, show users only the final answer unless the steps help them, and put the instruction in your system prompt so every request gets it.

When chain of thought is worth it, and when it is not

It tends to help on tasks with several dependent steps: sums, unit conversions, comparing options against rules, or planning a sequence. It adds little to tasks that are one step, like a short rewrite, a label or a translation of a sentence. There it mostly adds tokens.

It also does not fix every error. If the model reads a word as tokens it cannot split, as in why AI can't count letters, steps can help in some cases but the root cause is still there. When a number truly matters, have code do the arithmetic and let the model handle the wording.

Frequently asked questions

What is chain of thought prompting in simple terms?

It is asking an AI model to show its working before the answer. You either give solved examples with steps, or add an instruction such as "think step by step".

Does chain of thought prompting still help with newer models?

Often, but less predictably. The big gains were reported on 2022 models, and some newer models work through steps on their own, so test it on your own questions.

Is the chain of thought the model's real reasoning?

Not always. Studies from 2023 found written steps can leave out what actually influenced the answer, and that how much a model relies on its steps varies by task.

Does chain of thought cost more?

Yes. The steps are extra output tokens, so each request is slower and costs more. Measure the accuracy gain against that cost.

Get started

Chain of thought makes more sense once you see how a model predicts one token at a time. Sign in to Age of AI Entrepreneur with your email and an emailed code, open How Modern AI Actually Works, and follow its three short lessons, each with its sources.