Why ChatGPT Gives Different Answers to the Same Prompt

Why ChatGPT gives different answers to the same prompt comes down to one trick behind every chat assistant: next token prediction. The model reads the text so far, picks the next piece of text from a list of likely ones, adds it, and repeats. Because the pick involves chance, two runs can part ways. If you are a founder building with AI, this loop also explains why a model is fluent, why it is sometimes confidently wrong, and why your bill grows with the length of what you send.

The first lesson of the How Modern AI Actually Works course on Age of AI Entrepreneur is called "Next-Token Prediction", and a reader signs in with an email and an emailed code to open it. This post gives you the plain version first.

What a token is

A model does not read letters or whole sentences. It reads tokens: small chunks of text. A common word is often one token. A rare or long word is usually split into several pieces. Punctuation and spaces count too.

Before the model sees your message, software cuts it into tokens and turns each one into a number. After the model answers, the numbers are turned back into text. Everything in between works on numbers, one token at a time.

How the model picks each piece, step by step

Take the start of a sentence: "The customer asked for a". The model does not look up the answer. It scores every token it knows as a possible next piece, and turns the scores into chances. The picture below shows the idea with made-up numbers.

Then one token is picked from that list. The pick is not always the top one: a setting often called temperature controls how adventurous the choice is. A low setting stays close to the likeliest token. A high setting takes more chances, which gives more varied text and more mistakes.

The chosen token is added to the text, and the whole process runs again with the longer text. That is the loop:

A full answer is just this loop running a few hundred times. No plan is written first and then typed out. Each piece is chosen with the pieces before it in view.

Why one simple task produces useful answers

The model learned to score tokens by reading a huge amount of text and being corrected each time its guess was poor. To guess well, it had to pick up grammar, facts that appear often, the shape of an email, the structure of code. Nobody wrote those in as rules. They are by-products of getting the next piece right, over and over.

That raw model is not yet a helpful assistant. A second stage of training shapes it to follow instructions and answer in a useful way, which is the subject of the course's second lesson, "From Raw Model to Helpful Assistant". For how the big pieces fit together, see our post on AI, machine learning and deep learning, and for the network underneath, what a neural network is.

Why ChatGPT gives different answers, and what else follows

Four practical results follow straight from the loop.

• Fluent is not the same as true. The model picks what fits, not what it has checked. A wrong answer reads just as smoothly as a right one. Our post on real AI hallucination examples shows what that costs.
• Cost follows tokens. You pay for the tokens you send and the tokens that come back, so long prompts and long answers cost more. The "Model APIs, Tokens and What a Request Costs" lesson in The AI Product Stack course goes into this.
• Your words steer the guess. The text you send is the start of the sentence the model finishes. A clearer start gives a better finish.
• The same prompt can give different answers. Because a token is picked from a list of chances, one early difference changes every token after it. That is why you test with many examples and with repeat runs, not one.

A worked example: steering the next token

Say your product drafts replies to support emails. Compare two ways to start the same request.

1. Vague start: "Reply to this customer." The chances for the first tokens are spread wide, so you get a generic, chatty opening.
2. Clear start: "You write support replies for a small online shop. Use two short paragraphs. Start by saying what you will do about the problem. Customer email follows." Now the likely first tokens are an action, in a short, plain style.

You did not change the model. You changed the text the model is continuing, and so the chances for every token that follows. Now run both versions on twenty real emails, twice each, and read the results side by side: you see both the difference the wording makes and the difference between repeat runs. If you want a method for that, the Evals course at its course page has a lesson on building an eval set and judging outputs.

Where the simple story stops

"It only predicts the next token" is true as a description of the task. People still disagree about how much that implies. Some researchers argue that predicting text well forces a model to build something like an understanding of the world. Others argue that it only learns surface patterns in text and that fluent language is easy to mistake for understanding. Both groups agree on the mechanism; they differ on what it amounts to. The course lists its sources in every lesson, and the app sets out 17 cards on questions where researchers disagree, so you can read the arguments yourself.

For a founder, the safe working rule does not depend on who is right: treat the output as a strong guess, check it where mistakes are costly, and measure quality on your own examples.

Frequently asked questions

Why does ChatGPT give different answers to the same question?

Each piece of the answer is picked from a list of likely next tokens, not always the top one. One different pick early on changes everything written after it.

What is next token prediction?

It is the task of guessing the next small chunk of text, given the text so far. A language model does this again and again, adding each chunk to the text, until a full answer is written.

Is a token the same as a word?

Not exactly. A short common word is often one token, but a long or rare word is split into several tokens, and spaces and punctuation count as well.

Can I make the answers the same every time?

Lowering the temperature setting, where your model provider offers one, makes the output steadier, but it does not promise identical text every run. Test for the variation you can live with.

Does predicting the next token mean the model does not understand?

Researchers disagree. Some say good prediction requires a model of the world, others say it shows only patterns in text. What you can rely on is how it behaves on your own tests.

Get started

Open the How Modern AI Actually Works course for its three short lessons: Next-Token Prediction, From Raw Model to Helpful Assistant, and What AI Gets Wrong. You sign in with your email and an emailed code.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.