The full answer
Top-k and top-p sampling limit which tokens a language model may choose from at each step. Top-k keeps only the k most likely next tokens; top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p. Both cut off the long tail of unlikely tokens, so outputs stay varied without turning into nonsense.
Background: how a model picks the next token At each step the model gives a probability for every token in its vocabulary. Always taking the most likely one (greedy decoding) is predictable but often dull and repetitive. Sampling from the full distribution is varied but occasionally picks a very unlikely token that derails the text. Top-k and top-p sit between these.
- Sort tokens by probability, keep the top k, renormalise, and sample from them.
- k = 1 is greedy decoding; k = 50 allows much more variety.
- Weakness: a fixed k ignores how confident the model is. After "The capital of France is", almost all probability is on one token, yet k = 50 still allows 49 poor choices; in an open-ended sentence, 50 may be too few.
- Introduced for story generation by Fan and colleagues.
Top-p (nucleus) sampling
- Sort tokens by probability and keep adding them until their total reaches p, for example 0.9; sample from that set.
- The set adapts: when the model is confident it may hold one or two tokens; when many continuations are plausible, it holds many.
- Proposed by Holtzman and colleagues, who showed it produced more human-like text than top-k or pure sampling.
- p = 1.0 turns the filter off; lower values make output more focused.
Related settings
- Temperature reshapes the probabilities before the filter: low values sharpen them towards the most likely tokens, high values flatten them.
- Min-p, available in some open-model libraries, keeps tokens whose probability is at least a fraction of the top token's.
Using them in practice
| Goal | Typical setting |
|---|---|
| Extraction, classification, code | Low temperature, or default sampling with strict instructions |
| General assistant answers | Provider defaults |
| Brainstorming, creative writing | Higher temperature or top-p, for more variety |
- Change temperature or top-p, not both: they interact, and most providers advise adjusting one.
- Top-k is mainly exposed by open-model libraries; many hosted APIs offer only temperature and top-p.
- Many newer models, especially reasoning models, fix their sampling settings and reject other values, so check the API reference before relying on them.
- No setting makes output fully deterministic or correct; for reliability, rely on clear prompts, structured outputs and evaluation.
Last updated 16 October 2026