# What is top-p (nucleus) sampling, and how is it different from temperature?

Picking the next token at random from only the smallest set of likely tokens whose chances add up to p, so the model never picks from the long tail of unlikely words.

- Page: https://tecyfy.com/concepts/top-p-sampling
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Nucleus Sampling, Top-p

At each step a language model gives every [Token](https://tecyfy.com/concepts/token) in its vocabulary a probability of coming next. Top-p sampling keeps only the most likely tokens, adding them up from the top until their probabilities reach p, and picks the next token at random from that set. With p at 0.9, the model chooses among the tokens that together make up 90% of the probability and never from the thousands of unlikely ones after them.

## A worked example

Say a support assistant has written "Your order will arrive on" and the model's next-token probabilities are:

| Token | Probability | Running total |
| --- | --- | --- |
| Monday | 0.50 | 0.50 |
| Tuesday | 0.30 | 0.80 |
| Friday | 0.12 | 0.92 |
| time | 0.05 | 0.97 |
| Mars | 0.001 | 0.971 |

With p at 0.9, the set stops at Friday, the first token where the running total reaches 0.9. The model picks among Monday, Tuesday and Friday in proportion to their chances (Monday about 54% of the time, after the 3 are scaled to add up to 1), and "time" and "Mars" can never be chosen. If the model had been sure, with Monday at 0.95, the set would be Monday alone. That is the point of top-p: the number of candidates follows the model's confidence.

## Why not always pick the most likely token?

Picking the top token every time (greedy decoding) sounds safest, but on longer text it gets stuck repeating itself. Holtzman and colleagues showed in 2019 that text chosen to be as likely as possible reads as bland and repetitive, nothing like what people write, while sampling from the full distribution sometimes picks a wildly unlikely word and goes off the rails. Their fix, nucleus sampling, samples from the "nucleus" of the distribution and cuts off the unreliable tail.

## Top-p, top-k and temperature

- **[Top-k](https://tecyfy.com/concepts#top-k-sampling)** keeps a fixed number of tokens, say the top 50, however the probability is spread.
- **Top-p** keeps a varying number: a few when the model is sure, many when it is not. That is its advantage.
- **[Temperature](https://tecyfy.com/concepts/temperature)** reshapes the probabilities before either cut: lower makes the likely tokens likelier still, higher flattens them.

## Can you set it?

It depends on the model. Open models run with Hugging Face Transformers or vLLM take `top_p` directly. Some hosted models fix it: on the current Claude models, setting `top_p`, `top_k` or `temperature` to anything but the default returns an error, and you steer the answer with instructions, a schema and the effort setting instead.

## Related

- [Temperature](https://tecyfy.com/concepts/temperature): A setting that controls how much chance goes into each word a model picks. Low gives focused, predictable answers; high gives more varied ones.
- [Top‑k Sampling](https://tecyfy.com/concepts#top-k-sampling): A decoding strategy that restricts choices to the k most probable tokens
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.
- [Hallucination](https://tecyfy.com/concepts/hallucination): When a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.

## Sources

- [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751)
- [Generation strategies](https://huggingface.co/docs/transformers/main/en/generation_strategies)
- [Thinking](https://platform.claude.com/docs/en/build-with-claude/thinking)