# What is inference in AI?

Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.

- Page: https://tecyfy.com/concepts/inference
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Serving
- Last checked: 2026-10-14

A model has 2 lives. In the first it is trained: its billions of numbers are adjusted, again and again, until it predicts text well. That happens once, or once per version, and costs the maker a great deal. In the second it is used. Every question anyone sends it is a fresh run of the finished model. That second life is called inference.

During inference the model's numbers do not change. It takes your text, works through it, and produces a reply. Nothing you type teaches it anything new.

## Why it is slow at the end and fast at the start

A language model reads your whole prompt in one go, but it writes its answer one [Token](https://tecyfy.com/concepts/token) at a time, and each new token depends on everything before it. That is why a reply arrives as a stream of words. 2 numbers describe the wait. Time to first token is how long until the first word appears. Latency is the time until the whole answer is done. A bigger model, a busier server and a longer prompt all push both up.

## Why it is what you pay for

When you use a model through an API, you pay per token of inference: one price for the tokens you send and a higher one for the tokens it writes. On Claude Opus 5.5 that is 4 dollars per million tokens in and 20 dollars per million out. Training costs are the maker's problem. Inference costs are yours.

## How providers make it cheaper

Most of the engineering around serving models is about doing inference faster and with less memory.

- A [KV cache](https://tecyfy.com/concepts/kv-cache) keeps work from earlier tokens so it is not repeated for every new one.
- [Quantization](https://tecyfy.com/concepts/quantization) stores the model's numbers with less precision, so it fits on smaller hardware.
- Prompt caching lets a provider reuse the processing of a long prompt you send again and again, and charge you less for it.

When you see "inference" in a price list, a job advert or a chip maker's pitch, it means this second life: the model at work, answering.

## Related

- [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions.
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.
- [Latency](https://tecyfy.com/concepts#latency): The time between sending a request to a model and receiving the first token of its response.

## Sources

- [How ChatGPT and our foundation models are developed](https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed), OpenAI Help Center, 2026-09-25. In training the model's parameters are adjusted slightly over millions of sentences. A model is a large set of numbers called weights or parameters.
- [How caching works](https://huggingface.co/docs/transformers/main/en/cache_explanation), Hugging Face, Transformers documentation, 2026-10-14. Generation is slow because the model predicts one token at a time and each prediction depends on all the previous context. A key-value cache should only be used for inference.
- [Glossary](https://platform.claude.com/docs/en/about-claude/glossary), Anthropic, Claude API documentation, 2026-10-14. Latency is the delay between submitting a prompt and receiving the output. Time to first token measures how long the model takes to produce the first token of its output. Model size, hardware, network conditions and the length of the prompt and response affect both.
- [Pricing](https://platform.claude.com/docs/en/about-claude/pricing), Anthropic, Claude API documentation, 2026-09-25. Opus 5.5 costs 4 dollars per million input tokens and 20 dollars per million output tokens.
- [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), Anthropic, Claude API documentation, 2026-10-14. Prompt caching reduces processing time and costs for prompts with repeated content. Cache reads are priced at a fraction of the base input price.
- [Quantization overview](https://huggingface.co/docs/transformers/main/en/quantization/overview), Hugging Face, Transformers documentation, 2026-10-14. Quantization lowers the memory needed to load and use a model by storing the weights in a lower precision while trying to keep as much accuracy as possible.