What is inference in AI?

A model has 2 lives. In the first it is trained: its billions of numbers are adjusted, again and again, until it predicts text well. That happens once, or once per version, and costs the maker a great deal. In the second it is used. Every question anyone sends it is a fresh run of the finished model. That second life is called inference.

During inference the model's numbers do not change. It takes your text, works through it, and produces a reply. Nothing you type teaches it anything new.

Why it is slow at the end and fast at the start

A language model reads your whole prompt in one go, but it writes its answer one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at a time, and each new token depends on everything before it. That is why a reply arrives as a stream of words. 2 numbers describe the wait. Time to first token is how long until the first word appears. Latency is the time until the whole answer is done. A bigger model, a busier server and a longer prompt all push both up.

Why it is what you pay for

When you use a model through an API, you pay per token of inference: one price for the tokens you send and a higher one for the tokens it writes. On Claude Opus 5.5 that is 4 dollars per million tokens in and 20 dollars per million out. Training costs are the maker's problem. Inference costs are yours.

How providers make it cheaper

Most of the engineering around serving models is about doing inference faster and with less memory.

• A KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. keeps work from earlier tokens so it is not repeated for every new one.
• QuantizationQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. stores the model's numbers with less precision, so it fits on smaller hardware.
• Prompt caching lets a provider reuse the processing of a long prompt you send again and again, and charge you less for it.

When you see "inference" in a price list, a job advert or a chip maker's pitch, it means this second life: the model at work, answering.