# How LLMs work, for engineers: tokens, context windows, sampling and hallucinations

A large language model writes one token at a time. It reads everything in its context window, gives every possible next token a probability, picks one, adds it to the text and repeats. It remembers nothing between calls, so your code sends the whole conversation each time, and you pay for every token in and out. It was trained to give a fluent, helpful answer, not a verified one, which is why it can be confidently wrong and why the facts it needs should be in the prompt.

- Page: https://tecyfy.com/ai-engineering/how-llms-work
- Course: AI engineering course (https://tecyfy.com/ai-engineering), Module 1: How a model writes an answer

At the end of this module you can:

- Count your own text in tokens with the API, and turn the count into a price
- Explain how a model writes an answer one token at a time, and why output costs more than input
- Say what the context window holds, why a long conversation gets slower and dearer, and why the model forgets between calls
- Explain sampling, why the same prompt gives different answers, and which settings current models let you change
- Say how a base model becomes an assistant, and why that makes it confidently wrong

You will never train a model in this course, but every decision you make depends on how one writes. How long the prompt can be, why the bill grows, why the same question gets 2 answers, why the model makes up a returns policy: each comes from the same few mechanics. This module covers them at the level you use every day, and ends with a script that measures the shop's own text.

## From your words to tokens

A model does not read words or letters. A tokenizer first cuts your text into [tokens](https://tecyfy.com/concepts/token): short words, pieces of longer ones, spaces and punctuation, each mapped to a number. The model reads and writes those numbers. Everything you pay for and every limit you meet is counted in them: the price per million tokens, the size of the [context window](https://tecyfy.com/concepts/context-window), `max_tokens`, the rate limits.

How many tokens a text makes depends on the model's tokenizer, and tokenizers change between models. Anthropic says the tokenizer that came with [Claude Opus 4.7](https://tecyfy.com/prices/claude-opus-4-7) counts roughly 30% more tokens for the same text than the models before it. So do not guess from a word count, and do not reuse a count measured on another model. Ask the API:

```python
count = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Where is my order?"}],
)
print(count.input_tokens)
```

Counting is free and runs no model. It takes the same `system`, `messages` and `tools` you would send, so you can count a whole request before making it. The number is an estimate and can differ from the billed figure by a little. The Build it section at the end counts every help page of the shop this way.

- Tool: [Token counter](https://tecyfy.com/prices/token-counter). How many tokens your text is, on every model

## One token at a time

The models behind GenAI apps are [transformers](https://tecyfy.com/concepts/transformer). Each token is turned into a list of numbers, and [Self-Attention](https://tecyfy.com/concepts/self-attention) lets every token take in what is relevant from the tokens before it, so "bank" means something different next to "river" than next to "loan". The 2017 paper that introduced the design dropped the word-by-word processing of earlier networks, which made training far quicker on large amounts of text.

(An interactive demo goes here on the page: https://tecyfy.com/ai-engineering/how-llms-work)

Writing an answer is a loop. The model reads the whole context, gives every token in its vocabulary a probability of coming next, picks one, adds it to the text, and runs again with that token included. It stops when it writes an end token or reaches your `max_tokens`. 3 things follow for an engineer:

- **Output is slower and dearer than input.** Your prompt is read in one pass; the answer is written one token after another, each needing a pass of its own. On [Claude Opus 5.5](https://tecyfy.com/prices/claude-opus-5-5) an output token costs $20 per million and an input token $4, so a long answer can cost more than a long prompt.
- **Thinking is output too.** A model that reasons before answering writes those tokens in the same loop. They are billed as output and count toward `max_tokens`, which is why module 2 leaves room for them.
- **Each token is final.** Once chosen, a token becomes input for the rest of the answer. The model does not revise what it wrote; if you want a draft checked, that is another call, which module 7 builds as a judge.

## The context window

The context window is everything the model can see while it writes: your [system prompt](https://tecyfy.com/concepts/system-prompt), the conversation, any documents you added, the tool definitions, and the answer it is writing. Anthropic's docs call it the model's working memory, as opposed to what it learned in training. Current Claude models have a 1M-token window and can write up to 128k tokens in one reply.

(An interactive demo goes here on the page: https://tecyfy.com/ai-engineering/how-llms-work)

The model keeps nothing between calls. To continue a conversation your code sends every earlier message again, in full, and every one of them is billed as input every time. Turn 20 of a support chat pays for turns 1 to 19 once more. This is the single most common surprise on a first bill, and modules 6 and 9 show what to do about it: trim, summarise, and cache the part that does not change.

A bigger window is not a reason to fill it. Anthropic notes that as the token count grows, accuracy and recall degrade, which it calls context rot. A prompt with the 1 help page that answers the question usually does better than one with all 600, and it costs a small fraction as much. Choosing what goes into the window is most of modules 3 to 5.

At the edges, the API is strict. Input larger than the window fails with a 400 "prompt is too long" error. Input that fits, plus a `max_tokens` that would not, is accepted, and the answer stops with `stop_reason` set to `model_context_window_exceeded` if it reaches the limit. Some models also price long prompts differently: [Claude Haiku 5.5](https://tecyfy.com/prices/claude-haiku-5-5) charges 5 times as much per token for a prompt over 100,000 tokens.

## Sampling: why the same prompt gives different answers

A model's output at each step is a set of probabilities, and something has to choose. Always taking the most likely token sounds safest, but Holtzman and colleagues showed in 2019 that text built that way comes out "bland and strangely repetitive". So models sample: they pick at random, weighted by the probabilities. [Temperature](https://tecyfy.com/concepts/temperature) sharpens or flattens the probabilities before the pick, and [top-p](https://tecyfy.com/concepts/top-p-sampling) keeps only the likeliest tokens that together reach a share of the probability.

(An interactive demo goes here on the page: https://tecyfy.com/ai-engineering/how-llms-work)

Sampling is why the same prompt gives different answers, and why your tests become evals that run each case more than once (module 7). Anthropic notes that even a temperature of 0 does not make results fully identical.

What you can set depends on the model. On Claude Opus 5.5 and the other current Claude models, a `temperature`, `top_p` or `top_k` other than the default returns a 400 error. Open models you run yourself (module 10) still take all 3. Either way, temperature is a weak lever. Clear instructions, a JSON schema and a few examples make answers consistent far more reliably, and an exact rule, like "refunds only within 30 days", belongs in your code, where it cannot be sampled.

## How a base model becomes an assistant

A model starts with pre-training: it reads a very large amount of text and learns to predict the next token. The result, a base model, continues text. It was trained only to continue text, so given an instruction it may treat it as the start of a longer document and carry on writing that.

2 further steps turn it into the assistant you call:

1. **[Instruction tuning](https://tecyfy.com/concepts/instruction-tuning).** The model is fine-tuned on many tasks written as instructions with good answers. Google's FLAN study in 2021 showed that a 137 billion parameter model tuned on over 60 such tasks got much better at kinds of task it had never been tuned on, beating the larger GPT-3, asked without examples, on 20 of 25.
2. **[Reinforcement learning from human feedback](https://tecyfy.com/concepts/reinforcement-learning-from-human-feedback).** People rank the model's answers, a reward model learns their preferences, and the model is tuned to score well. In OpenAI's 2022 InstructGPT work, answers from a 1.3 billion parameter model trained this way were preferred to those of the 175 billion parameter GPT-3.

The lesson in that result is the one to keep: size alone did not make a model follow instructions; training on what people preferred did. The assistant you call was trained to be helpful to whoever writes the instruction, and it sees every instruction in its context, which is useful when the text is yours and risky when it came from a stranger's email (module 8).

Training stops at a date, the model's [knowledge cutoff](https://tecyfy.com/concepts/knowledge-cutoff): June 2026 for the current Claude models. Anything newer, and anything private, such as your shop's prices, it can only know if you put it in the prompt.

- Lesson: [How AI learns: a guessing game, not a database](https://tecyfy.com/learn/how-ai-works/how-ai-learns). An AI model learned by guessing the next word across a vast pile of text, and what it kept is patterns, not pages. This lesson follows that training in order,…

## Why a model is confidently wrong

Ask the model for the shop's returns window without giving it the returns page, and it will often answer anyway, fluently and with no sign of doubt. That is a [Hallucination](https://tecyfy.com/concepts/hallucination): a plausible answer with nothing behind it.

It is not a bug in one model. A 2025 OpenAI study argues that training and evaluation both reward guessing: a model scored on right answers, with no credit for "I don't know", does best by always guessing, like a student on a multiple-choice exam with no penalty for wrong answers. Fluency and confidence say nothing about whether an answer is true.

What works, in the order this course builds it:

- **Give it the facts.** Put the relevant help page in the prompt ([RAG](https://tecyfy.com/concepts/rag), modules 4 and 5), so it answers from text rather than memory.
- **Allow it to decline.** Tell it to answer only from the pages and to say when they do not cover the question, then check that it does.
- **Measure.** An [eval set](https://tecyfy.com/concepts/eval-set) with questions the pages do not answer shows how often it guesses anyway (module 7).
- **Keep exact facts in code.** An order's status comes from your database through a tool, not from the model (module 6).

## What you set, and what it costs to get wrong

| You set | What it does | What goes wrong |
| --- | --- | --- |
| `model` | Which model, its price, window and tokenizer | A count or budget measured on another model is wrong |
| `max_tokens` | The most the reply, thinking included, may run to | Too low and the answer is cut off mid-JSON (`stop_reason` is `max_tokens`) |
| `system` and `messages` | Everything the model sees | Too little and it guesses; too much and it costs more and recalls less |
| `output_config.effort` | How much the model thinks before answering | Higher effort means more output tokens to pay for and wait on |
| `temperature`, `top_p` | Sampling, on models that allow it | A 400 error on current Claude models |

- Lesson: [How AI writes its answers: one piece at a time](https://tecyfy.com/learn/how-ai-works/how-ai-writes). An AI chat app does not work out a whole answer and then show it. It writes one small piece at a time, and a weighted roll of the die picks each piece. This…

## Build it: measure the shop's text in tokens

The support assistant you build in this course answers from the shop's 6 help pages, in a `docs/` folder of Markdown files. Before deciding how to use them, measure them. `tokens.py` does 3 things:

1. counts each help page in tokens and compares it with its word count,
2. prices the simplest design, every page in every prompt, on 2 models, and checks how much of the window it takes,
3. holds a 3-turn conversation and prints the input tokens of each turn.

```python
"""Measure the shop's text in tokens: what each help page costs to send, and how a conversation grows."""
from pathlib import Path

import anthropic

MODEL = "claude-opus-5-5"
PRICES = {"claude-opus-5-5": (4.00, 20.00), "claude-haiku-5-5": (0.10, 0.50)}  # dollars per million tokens

client = anthropic.Anthropic()


def tokens(text: str, model: str = MODEL) -> int:
    """Input tokens for one user message, counted by the API for this model. Free, and no model runs."""
    return client.messages.count_tokens(model=model, messages=[{"role": "user", "content": text}]).input_tokens


# 1. Words are not tokens: count each help page.
pages = sorted(Path("docs").glob("*.md"))
for page in pages:
    text = page.read_text()
    n = tokens(text)
    print(f"{page.name:<14} {len(text.split()):>4} words  {n:>5} tokens  {len(text) / n:.1f} characters a token")

# 2. Put every page in every prompt? Price it before deciding.
everything = "\n\n".join(page.read_text() for page in pages)
email = "My mixer was meant to arrive Monday. Where is it? Order KX-48213."
windows = {model.id: model.max_input_tokens for model in client.models.list()}
print()
for model, (input_price, _) in PRICES.items():
    n = tokens(f"{everything}\n\n<email>\n{email}\n</email>", model)
    per_email = n * input_price / 1_000_000
    print(f"{model:<17} {n:>6} input tokens, {n / windows[model]:.2%} of its window, "
          f"${per_email:.5f} an email, ${per_email * 10_000:.2f} for 10,000 emails, before any output")

# 3. The model remembers nothing between calls: each turn sends the whole conversation again.
conversation = []
print()
for turn, question in enumerate(["Where is order KX-48213?", "Can I change the delivery address?",
                                 "And if it arrives late, can I get a refund?"], start=1):
    conversation.append({"role": "user", "content": question})
    response = client.messages.create(model="claude-haiku-5-5", max_tokens=300, messages=conversation)
    conversation.append({"role": "assistant", "content": response.content})
    print(f"turn {turn}: {response.usage.input_tokens:>4} tokens in, {response.usage.output_tokens:>4} out")
```

Parts 1 and 2 cost nothing, since counting is free; part 3 makes 3 small calls on Claude Haiku 5.5. The prices are in the script by hand, from Anthropic's pricing page, so check them against [the current prices](/prices) when you run it.

Run it with `python tokens.py`. This is its output against a stand-in for the API that estimates tokens from characters, so your numbers will differ; the shape will not:

```text
cookware.md     151 words    223 tokens  3.7 characters a token
delivery.md     110 words    170 tokens  3.6 characters a token
knives.md        87 words    131 tokens  3.6 characters a token
mixer.md        119 words    168 tokens  3.6 characters a token
returns.md      100 words    146 tokens  3.6 characters a token
warranty.md      97 words    150 tokens  3.6 characters a token

claude-opus-5-5      956 input tokens, 0.10% of its window, $0.00382 an email, $38.24 for 10,000 emails, before any output
claude-haiku-5-5     956 input tokens, 0.10% of its window, $0.00010 an email, $0.96 for 10,000 emails, before any output

turn 1:   14 tokens in,   15 out
turn 2:   77 tokens in,   15 out
turn 3:  142 tokens in,   15 out
```

Read it as an engineer would:

- **Tokens outnumber words.** Every page makes more tokens than words, so a budget planned in words comes out short.
- **6 pages fit easily, and still cost something.** They use a tenth of 1% of the window, so for this shop, sending them all is a real option. The same choice on a shop with 600 pages would be 100 times the tokens on every email, and the price difference between models, about 40 times here, would decide the design. Module 4 adds search so the prompt carries only the pages that matter.
- **The conversation grows.** Each turn's input includes every turn before it. Add a fourth and fifth question and watch the first number climb.

Try 3 changes before moving on. Count the same page with `claude-haiku-5-5` and `claude-opus-5-5` and compare. Add a system prompt to the counting call and see what it adds. And add the help pages to the first message of the conversation in part 3, then watch every later turn pay for them again; that is the cost module 9's caching removes.

## Check yourself

1. A customer chats with your support assistant for 20 turns. Each turn's reply is about the same length, but the bill for turn 20 is far higher than for turn 1. Why?

   A. The model writes longer hidden thinking as a conversation goes on
   B. The API charges a higher rate per token after the first few turns
   C. The model is slower later in a conversation, and time is billed
   D. Every call sends the whole conversation so far, so the input grows with each turn

2. Your team measured the help pages at 40,000 tokens last year on an older Claude model, and plans the budget for a newer model on that figure. What should you do first?

   A. Count the pages again with the token counting endpoint, passing the model you plan to use
   B. Use the old figure, since a token is a fixed unit of text
   C. Divide the word count by 4, which gives the tokens on any model
   D. Add 10% to the old figure to be safe

3. Asked for the shop's returns window, your assistant replies "60 days" with full confidence. The real policy is 30 days, and it was not in the prompt. What is the most useful fix?

   A. Set the temperature to 0 so the model stops guessing
   B. Ask the model to double-check its answer before replying
   C. Put the returns page in the prompt, tell the model to answer only from it and to say it does not know otherwise, and test that with an eval
   D. Switch to the largest model, which does not make things up

### Answers

1. D. The model keeps no memory between calls. To continue a conversation your code sends every earlier message again, and each of them is billed as input every time. The rate per token does not rise with turns, and time is not billed. This is why long conversations are trimmed, summarised or cached.
2. A. A token is whatever piece of text a model's tokenizer makes, and tokenizers change. Anthropic says the tokenizer that came with Claude Opus 4.7 counts roughly 30% more tokens for the same text than the models before it, and tells you to count with the model you plan to use. The counting endpoint is free, so there is no reason to estimate.
3. C. The model was never told the policy, so it wrote a plausible answer; training rewards a fluent guess over "I don't know". Giving it the facts and an explicit way to decline fixes the cause, and an eval shows whether the fix holds. Temperature does not add knowledge, and current Claude models reject a non-default temperature anyway. A larger model guesses less often, but still guesses.

## Questions

### How does a large language model generate text?

It works one token at a time. The model reads everything in its context window, gives every token in its vocabulary a probability of coming next, picks one, adds it to the text and repeats until it decides to stop or reaches the max_tokens limit. Each new token is fed back in as input for the next.

### What is a token in an LLM?

A token is the unit of text a model reads and writes, often a short word or part of a longer one, with spaces and punctuation included. On Claude, count tokens with the free count_tokens endpoint for the model you will use, since the same text counts differently on different models. You are billed per token, in and out.

### Does an LLM remember previous conversations?

No. A model keeps nothing between API calls. Your code sends the conversation so far with every request, and the model reads it all again each time. That is also why a long conversation costs more on every turn. Chat apps that seem to remember are storing and resending text for you.

### Why do LLMs hallucinate?

A model is trained to write likely, fluent text, and its training and the tests it is scored on reward a confident guess over saying it does not know. When the answer is not in its context or its training, it writes the most plausible one. Put the facts in the prompt, allow it to say it does not know, and measure with evals.

### Can I make an LLM give the same answer every time?

Not fully. Sampling picks tokens at random from the model's probabilities, and even a temperature of 0 does not make answers identical. On current Claude models you cannot change temperature at all. Make answers consistent with clear instructions, a JSON schema and examples, and treat your code, not the model, as the place where exact rules live.

## Sources

- [Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)
- [Context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows)
- [Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
- [Thinking](https://platform.claude.com/docs/en/build-with-claude/thinking)
- [Models overview](https://platform.claude.com/docs/en/about-claude/models/overview)
- [Glossary](https://platform.claude.com/docs/en/about-claude/glossary)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751)
- [Finetuned Language Models Are Zero-Shot Learners](https://arxiv.org/abs/2109.01652)
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
- [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664)