Is ChatGPT Just Autocomplete? Where the Debate Stands

Is ChatGPT just autocomplete? The honest answer is "autocomplete is where it starts, and not where it ends". A chat assistant is built on a model trained to predict the next token of text, but other training and a very large scale sit on top of that. If you plan to build a product on one, knowing which half of the claim is true helps you decide what to trust it with.

The How Modern AI Actually Works course on Age of AI Entrepreneur opens with this question in its first lesson, Next-Token Prediction, and a reader signs in with an email and an emailed code to open it. This post follows the same path with a founder's questions in mind.

Is ChatGPT just autocomplete? The short answer

Part of the claim is correct. The model behind a chat assistant is a language model, and the best-known paper on one of them, the GPT-3 paper, describes it as an autoregressive language model: it produces text one piece at a time, each piece chosen given the pieces before it. That is the mechanism the word "autocomplete" points at.

Part of the claim leaves things out. Your phone keyboard suggests a word and stops. A large language model is trained at a far bigger scale, then trained again to follow instructions, and it can be given a task and examples as plain text. Whether those extras amount to something more than autocomplete is the part researchers still argue about.

What next-token prediction actually does

A token is a small chunk of text: a word, part of a word or a punctuation mark. During training, the model reads huge amounts of text and keeps adjusting itself so that, given the tokens so far, it gives a higher score to the token that really came next.

When you use it, it does the same thing in a loop. It reads your prompt, scores every possible next token, picks one, adds it to the text and repeats. The sentence you read is a chain of those choices. This is also why the same prompt can give different answers: the pick is not always the top-scored token, a point our post on why ChatGPT gives different answers to the same prompt works through.

Three layers between the predictor and the assistant

The picture below puts the layers in order. The first is the pretrained predictor. The second is what turns it into something you can chat with. The third is you.

Layer 1, pretraining. The GPT-3 paper describes a model with 175 billion parameters. Its authors tested it in a few-shot setting, where the task and a few examples are given purely as text, with no gradient updates or fine-tuning.

Layer 2, post-training. The InstructGPT paper reports that making a language model bigger does not by itself make it better at following what a user wants. Large models can be untruthful, toxic or unhelpful. The authors fine-tuned GPT-3 with human demonstrations and then with reinforcement learning from human feedback, and labelers preferred the outputs of the 1.3 billion parameter InstructGPT to those of the 175 billion parameter GPT-3. Our post on how chatbots learn to be helpful goes through that method.

Layer 3, your prompt. What you write is the only thing the model sees of your job. It reads your instructions as text, like everything else.

Where researchers disagree

One strong version of "just autocomplete" says nothing happens beyond pattern matching on text. A strong version of the other side says that scale produces new abilities. The research does not settle either claim, and a sourced example shows why.

• Abilities that appear with scale. The paper "Emergent Abilities of Large Language Models" describes an ability as emergent if larger models have it and smaller ones do not, so you cannot predict it by extending the trend from smaller models. Its authors argue that this suggests more scaling could expand what these models can do.
• A different reading of the same results. The paper "Are Emergent Abilities of Large Language Models a Mirage?" argues that the sudden jumps often come from the choice of metric. Nonlinear or discontinuous metrics make performance look abrupt, while linear or continuous ones show gradual, predictable improvement. It concludes that emergence may not be a fundamental property of scaling.
• Not in dispute. The models are trained by next-token prediction and then adjusted by feedback, and the GPT-3 authors themselves report that some datasets remain difficult for the model.

The course gives such questions the same treatment: each lesson lists its sources, and cards set out where researchers disagree, without picking a side for you.

A worked example: a 20-minute test you can run

You do not need to settle the debate to decide what to trust. Test the assistant you plan to use on three jobs and write down what you see.

1. Minute 0 to 5, a continuation. Paste the start of a sentence and ask for one word to continue it. Run it three times and write the three answers.
2. Minute 5 to 12, an example in the prompt. Give two made-up examples of a rule, such as turning a product name into a one-line tagline in a style you invent, and ask for a third. You are checking whether an example given as text is enough for it to follow the pattern.
3. Minute 12 to 20, a check you can verify. Ask for something with a right answer you can compute yourself, such as the number of letters in a word or a simple sum. Note each wrong answer and whether it said it was unsure.

Put the results in a three-column note: job, what you asked, what you got. If step 3 fails, that tells you something no argument about autocomplete can: where your product needs a check of its own before a customer sees the answer. Building that habit is the subject of the lesson Building an Eval Set and Judging Outputs in the Evals: Shipping AI That Works course.

What this means for your product

The label matters less than three practical points.

• Fluent is not the same as correct. A model that chooses likely tokens will write likely-sounding text. The third lesson of the course, What AI Gets Wrong, and our post on AI hallucination examples cover what that looks like.
• The model is a base you adapt. You rarely train one. See our guide to what a foundation model is.
• Test on your own examples. Whatever the right word for the mechanism, only your own test cases tell you whether it does your job.

Frequently asked questions

Is ChatGPT only predicting the next word?

Its base model is trained to predict the next token, and it generates text one token at a time. It was also fine-tuned with human feedback, which the InstructGPT paper describes, so it does more than continue your text.

Does predicting the next token mean it understands nothing?

Researchers do not agree on this, and it depends on what "understand" means. The papers cited above debate what scale adds, and this post takes no side.

Why does the same prompt give different answers?

The model scores possible next tokens, and the choice among them is not always the top one. Settings that control that choice change how varied the answers are.

Can I build a product if I do not know how the model works inside?

Yes. Test it on your own examples and add a check where a wrong answer would cost you. The first lessons of the course teach the mechanism in about nine minutes each.

Get started

Open Age of AI Entrepreneur, sign in with your email and an emailed code, and start with the How Modern AI Actually Works course. Each lesson takes about nine minutes and lists its sources, and the terms page says every lesson is free to anyone with an account.