What Is Top-P Sampling? How an AI Picks Its Next Word

What is top-p sampling? It is a rule a language model uses to choose its next word. The model ranks every possible next token by how likely it is, keeps the smallest group of top choices whose chances add up to a number p (such as 0.9), throws away the rest, and then picks at random from the group it kept. A low p keeps only the safest words. A high p lets rarer words in.

Top-p is one small part of next-token prediction, the first lesson of the How Modern AI Actually Works course on Age of AI Entrepreneur, a Basic course of three short lessons for founders with no background; sign in with your email and an emailed code to open it.

What is top-p sampling, step by step

A language model writes one token at a time. A token is a word or a piece of a word. At each step the model produces a score for every token it knows, and those scores become probabilities that add up to 1. The question is which token to actually write.

Top-p answers it in four steps:

1. Sort. Line up every candidate token from most likely to least likely.
2. Add up. Walk down the list, keeping a running total of the probabilities.
3. Cut. Stop as soon as the running total reaches p. Everything below that point is dropped.
4. Draw. Pick one of the kept tokens at random, in proportion to its chance.

The kept group is called the nucleus, which is why top-p is also called nucleus sampling. The method comes from a 2019 research paper, "The Curious Case of Neural Text Degeneration" by Holtzman and colleagues. Its abstract describes sampling from "the dynamic nucleus of the probability distribution", which keeps variety in the text while cutting off "the less reliable tail".

Why a model needs a sampling rule at all

You might think the model should always write its single most likely token. That approach exists and is called greedy decoding. It is predictable, and for short factual answers it can be the right choice. For longer text, the paper above reports that always choosing the most likely words tends to produce bland and repetitive writing.

The opposite approach samples from the full list, every token in proportion to its chance. That brings variety, but it also gives the long tail of unlikely tokens a chance on every single step. Over hundreds of steps, one odd word can send the whole answer off course.

Top-p sits between the two. It keeps the randomness that makes text read naturally, but only among tokens the model already rates as reasonable. This is also one of the reasons ChatGPT gives different answers to the same prompt: with sampling switched on, two runs can take different paths from the very first token.

A worked example you can follow

Say your support tool is writing a shipping update, and the text so far is "Thanks for your order. Your package will arrive on". The chart below uses made-up probabilities to show the cutoff at p = 0.9.

Walk through it. "Monday" has 0.50, so the running total is 0.50. "Tuesday" adds 0.25, making 0.75. "time" adds 0.10, making 0.85. "Friday" adds 0.06, making 0.91, which passes 0.9, so the list stops there. "schedule" and "banana" are cut and can never be written at this step.

Now the model draws from Monday, Tuesday, time and Friday, weighted by their chances. Most runs write "Monday". Some write "Tuesday". Now and then you get "Friday". You never get "banana".

Change p and the group changes. At p = 0.5 only "Monday" survives, so the output is the same every time. At p = 1.0 nothing is cut, and even "banana" has its one in a hundred chance.

Notice what the nucleus does when the model is unsure. If the probabilities were spread thin across dozens of words, reaching 0.9 would take many of them, so the kept group grows. When one word is a near certainty, the group shrinks to that word. That is the "dynamic" part: the cutoff adapts to how confident the model is at each step.

Top-p vs temperature: two dials for one choice

Most model APIs offer a second setting, temperature, and the two are easy to mix up. Temperature changes the shape of the probabilities before any cut. A low temperature makes the top token even more dominant, and a high temperature flattens the list so rarer tokens get closer to the leaders. Top-p then decides where to cut that list.

Put simply, temperature changes how the chances are spread, and top-p changes which tokens are allowed into the draw at all. Our guide to LLM temperature walks through that setting with its own example.

Because both dials push in the same direction, changing them together makes it hard to tell which one caused a change in your output. A common working habit is to move one and leave the other at its default while you test.

How to set top-p for your own product

There is no single right value. It depends on whether you want the same answer every time or fresh wording each time. Here is a test you can run in an afternoon:

1. Pick one real task from your product, such as a shipping update or a product description.
2. Write 10 test inputs that look like what your users send.
3. Run each input 3 times at a low top-p (for example 0.5) and 3 times at a high one (for example 0.95), with temperature left at its default.
4. For each output, mark two things: is every fact right, and does it read well?
5. Count the marks for each setting. Keep the setting that wins on the mark that matters most for this task.

For tasks where facts and format must not change, such as extracting an order number or filling a form, a low top-p or greedy decoding often fits. For tasks where variety is the point, such as headline ideas or marketing drafts, a higher top-p gives you more to choose from. Write the result down with the date and model name, because a new model version can shift its probabilities.

What top-p cannot fix

Top-p only chooses among tokens the model already rates as likely. If the model rates a wrong fact as likely, a low top-p will write that wrong fact more reliably, not less. The setting controls randomness, not truth.

That is the theme of the course's third lesson, What AI Gets Wrong. A model can be fluent and confident and still be wrong, as these AI hallucination examples show. The fixes sit elsewhere: give the model the source text it needs, test outputs before release, and keep a person on the steps where a mistake is costly.

The same course covers what happens between a raw next-token predictor and the helpful assistant you chat with, in its second lesson, From Raw Model to Helpful Assistant. Each lesson lists its sources, and you answer questions as you go: choose, put in order, match, sort and estimate a number.

Frequently asked questions

Is top-p the same as nucleus sampling?

Yes. The kept group of tokens is called the nucleus, so top-p sampling and nucleus sampling name the same method, introduced in the 2019 paper by Holtzman and colleagues.

What does top-p = 1 mean?

Nothing is cut. Every token stays in the draw with its full chance, so the output is shaped only by temperature and the model's own probabilities.

What is the difference between top-p and top-k?

Top-k keeps a fixed number of tokens, such as the 40 most likely, at every step. Top-p keeps however many tokens it takes to reach the probability p, so the group grows when the model is unsure and shrinks when it is confident.

Does a lower top-p stop hallucinations?

No. It makes the output more predictable, but if the model rates a false statement as likely, a lower top-p can repeat it more often. Checking outputs against sources is what catches those errors.

Get started

Next-token prediction, the step where top-p does its work, is the first of three lessons in the How Modern AI Actually Works course on Age of AI Entrepreneur. The lessons take about 9 minutes each, every attempt is graded, and you can retake a lesson any time. Questions from lessons you finish come back later, spaced out, up to 5 a day, so the ideas stay with you. Sign in with your email and an emailed code to open the course.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.