# What is speculative decoding, and why does it make a model answer faster?

A small, fast drafter guesses the next few tokens and the large model checks them all in one pass, keeping the ones it agrees with. Answers come faster and stay exactly the same.

- Page: https://tecyfy.com/concepts/speculative-decoding
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: Speculative sampling, Draft model, Multi-token prediction, MTP

A language model writes one [Token](https://tecyfy.com/concepts/token) at a time, and for each token it reads all of its weights from memory. On a single conversation the chip spends most of that time waiting for memory, not computing. Speculative decoding uses that idle compute.

## How it works

A drafter, much cheaper than the main model, proposes the next few tokens, say 5. The main model then runs once over all 5 at the same time, which costs about as much as writing 1, because the weights are read once for the whole batch. Wherever the main model would have written the same token, the draft is kept. At the first token it disagrees with, it writes its own choice there and the rest of the draft is thrown away. Then the drafter guesses again from that point.

If the drafter guessed 3 of 5 right, the main model produced 4 tokens (3 kept plus its own correction) for the price of about 1 pass. If it guessed none right, nothing is lost but the drafter's small cost. The method's authors at Google showed that, with the right acceptance rule, the answer is exactly what the main model alone would have written, token for token, and they measured 2 to 3 times faster decoding.

## Where the drafter comes from

- **A separate small model** from the same family, sharing the tokenizer.
- **Extra heads on the main model** that predict several tokens ahead, called multi-token prediction (MTP). NVIDIA's Nemotron 3.5 Super VL is trained with one, so it needs no second model.
- **A small trained drafter** built for one model, such as EAGLE, or the DFlash and LiLiCorr drafters NVIDIA publishes next to Nemotron 3.5 Bolt.
- **No model at all**: n-gram and suffix methods copy tokens that already appear in the prompt, which works well when the answer repeats its input, as in code edits.

In [vLLM](https://tecyfy.com/concepts/vllm) you switch it on with `--speculative-config`, naming the method, the drafter and how many tokens to guess.

## When it helps and when it does not

It helps most where one person waits for one answer: a chat, a coding agent, a voice assistant on a local GPU. vLLM's documentation says the same: the gain is largest at low traffic, where the GPU is bound by memory. Under heavy batched load the GPU is already busy, so the extra checking work eats into throughput, and some methods help less there.

It costs memory too. A separate drafter needs its own weights and its own [KV cache](https://tecyfy.com/concepts/kv-cache), which counts when a model is sized to fill a card. NVIDIA's local recipes for Bolt run the drafter with a single conversation at a time.

## Related

- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.
- [Inference Latency](https://tecyfy.com/concepts/inference-latency): The time a model takes to produce an answer, split into reading the prompt (prefill, which sets the time to first token) and writing the answer one token at a time (decode).
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.
- [Knowledge Distillation](https://tecyfy.com/concepts/knowledge-distillation): Training a small "student" model to copy the outputs of a large "teacher" model, so the small one keeps much of the large one's skill at a fraction of the cost to run.

## Sources

- [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
- [Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/)
- [NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16)
- [NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16)