How LLMs work, for engineers: tokens, context windows, sampling and hallucinations

You will never train a model in this course, but every decision you make depends on how one writes. How long the prompt can be, why the bill grows, why the same question gets 2 answers, why the model makes up a returns policy: each comes from the same few mechanics. This module covers them at the level you use every day, and ends with a script that measures the shop's own text.

From your words to tokens

A model does not read words or letters. A tokenizer first cuts your text into tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.: short words, pieces of longer ones, spaces and punctuation, each mapped to a number. The model reads and writes those numbers. Everything you pay for and every limit you meet is counted in them: the price per million tokens, the size of the context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out., max_tokens, the rate limits.

How many tokens a text makes depends on the model's tokenizer, and tokenizers change between models. Anthropic says the tokenizer that came with Claude Opus 4.7 counts roughly 30% more tokens for the same text than the models before it. So do not guess from a word count, and do not reuse a count measured on another model. Ask the API:

count = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Where is my order?"}],
)
print(count.input_tokens)

Counting is free and runs no model. It takes the same system, messages and tools you would send, so you can count a whole request before making it. The number is an estimate and can differ from the billed figure by a little. The Build it section at the end counts every help page of the shop this way.

ToolToken counterHow many tokens your text is, on every model

One token at a time

The models behind GenAI apps are transformersTransformerThe design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.. Each token is turned into a list of numbers, and Self-AttentionSelf-AttentionThe step in a transformer where every token looks at the other tokens in the same text and takes in what is relevant to it, so meaning depends on context. lets every token take in what is relevant from the tokens before it, so "bank" means something different next to "river" than next to "loan". The 2017 paper that introduced the design dropped the word-by-word processing of earlier networks, which made training far quicker on large amounts of text.

Try itSee what a word pays attention to

The last word

bigsmall

1. The
2. trophy
3. did
4. not
5. fit
6. in
7. the
8. case
9. because
10. it
11. was
12. too
13. big

1. The2%
2. trophy43%
3. did2%
4. not2%
5. fit2%
6. in2%
7. the2%
8. case11%
9. because2%
10. it9%
11. was2%
12. too2%
13. big17%

Writing an answer is a loop. The model reads the whole context, gives every token in its vocabulary a probability of coming next, picks one, adds it to the text, and runs again with that token included. It stops when it writes an end token or reaches your max_tokens. 3 things follow for an engineer:

• Output is slower and dearer than input. Your prompt is read in one pass; the answer is written one token after another, each needing a pass of its own. On Claude Opus 5.5 an output token costs $20 per million and an input token $4, so a long answer can cost more than a long prompt.
• Thinking is output too. A model that reasons before answering writes those tokens in the same loop. They are billed as output and count toward max_tokens, which is why module 2 leaves room for them.
• Each token is final. Once chosen, a token becomes input for the rest of the answer. The model does not revise what it wrote; if you want a draft checked, that is another call, which module 7 builds as a judge.

The context window

The context window is everything the model can see while it writes: your system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say., the conversation, any documents you added, the tool definitions, and the answer it is writing. Anthropic's docs call it the model's working memory, as opposed to what it learned in training. Current Claude models have a 1M-token window and can write up to 128k tokens in one reply.

Try itFill a context window

Window

8,000 tokens16,000

When it fills

Drop the oldestSummarise

The window7,000 / 8,000 tokens

• App instructions
• Turn 1, with the flight date
• Later turns

Out of view

• Turn 1: “My flight is on 12 March”

Input tokens sent so far

27,200

over 5 turns

Send another turnAsk: when is my flight?New chat

The model keeps nothing between calls. To continue a conversation your code sends every earlier message again, in full, and every one of them is billed as input every time. Turn 20 of a support chat pays for turns 1 to 19 once more. This is the single most common surprise on a first bill, and modules 6 and 9 show what to do about it: trim, summarise, and cache the part that does not change.

A bigger window is not a reason to fill it. Anthropic notes that as the token count grows, accuracy and recall degrade, which it calls context rot. A prompt with the 1 help page that answers the question usually does better than one with all 600, and it costs a small fraction as much. Choosing what goes into the window is most of modules 3 to 5.

At the edges, the API is strict. Input larger than the window fails with a 400 "prompt is too long" error. Input that fits, plus a max_tokens that would not, is accepted, and the answer stops with stop_reason set to model_context_window_exceeded if it reaches the limit. Some models also price long prompts differently: Claude Haiku 5.5 charges 5 times as much per token for a prompt over 100,000 tokens.

Sampling: why the same prompt gives different answers

A model's output at each step is a set of probabilities, and something has to choose. Always taking the most likely token sounds safest, but Holtzman and colleagues showed in 2019 that text built that way comes out "bland and strangely repetitive". So models sample: they pick at random, weighted by the probabilities. TemperatureTemperatureA setting that controls how much chance goes into each word a model picks. Low gives focused, predictable answers; high gives more varied ones. sharpens or flattens the probabilities before the pick, and top-pTop-p SamplingPicking the next token at random from only the smallest set of likely tokens whose chances add up to p, so the model never picks from the long tail of unlikely words. keeps only the likeliest tokens that together reach a share of the probability.

Try itTurn the temperature up and down

Temperature1.0

The cat sat on the …

• mat50%
• sofa22%
• floor17%
• roof7%
• keyboard4%
• moon1%

Pick the next word

Sampling is why the same prompt gives different answers, and why your tests become evals that run each case more than once (module 7). Anthropic notes that even a temperature of 0 does not make results fully identical.

What you can set depends on the model. On Claude Opus 5.5 and the other current Claude models, a temperature, top_p or top_k other than the default returns a 400 error. Open models you run yourself (module 10) still take all 3. Either way, temperature is a weak lever. Clear instructions, a JSON schema and a few examples make answers consistent far more reliably, and an exact rule, like "refunds only within 30 days", belongs in your code, where it cannot be sampled.

How a base model becomes an assistant

A model starts with pre-training: it reads a very large amount of text and learns to predict the next token. The result, a base model, continues text. It was trained only to continue text, so given an instruction it may treat it as the start of a longer document and carry on writing that.

2 further steps turn it into the assistant you call:

1. Instruction tuningInstruction TuningFine-tuning a pre-trained model on many tasks written as instructions with good answers, so it follows a request instead of just continuing the text.. The model is fine-tuned on many tasks written as instructions with good answers. Google's FLAN study in 2021 showed that a 137 billion parameter model tuned on over 60 such tasks got much better at kinds of task it had never been tuned on, beating the larger GPT-3, asked without examples, on 20 of 25.
2. Reinforcement learning from human feedbackReinforcement Learning from Human FeedbackA training step where people rank a model's answers, a reward model learns those preferences, and the language model is tuned to score well on it.. People rank the model's answers, a reward model learns their preferences, and the model is tuned to score well. In OpenAI's 2022 InstructGPT work, answers from a 1.3 billion parameter model trained this way were preferred to those of the 175 billion parameter GPT-3.

The lesson in that result is the one to keep: size alone did not make a model follow instructions; training on what people preferred did. The assistant you call was trained to be helpful to whoever writes the instruction, and it sees every instruction in its context, which is useful when the text is yours and risky when it came from a stranger's email (module 8).

Training stops at a date, the model's knowledge cutoffKnowledge CutoffThe date a model's training text ends. The model knows nothing after it unless it is given new text, for example by a web search.: June 2026 for the current Claude models. Anything newer, and anything private, such as your shop's prices, it can only know if you put it in the prompt.

LessonHow AI learns: a guessing game, not a databaseAn AI model learned by guessing the next word across a vast pile of text, and what it kept is patterns, not pages. This lesson follows that training in order,…

Why a model is confidently wrong

Ask the model for the shop's returns window without giving it the returns page, and it will often answer anyway, fluently and with no sign of doubt. That is a HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.: a plausible answer with nothing behind it.

It is not a bug in one model. A 2025 OpenAI study argues that training and evaluation both reward guessing: a model scored on right answers, with no credit for "I don't know", does best by always guessing, like a student on a multiple-choice exam with no penalty for wrong answers. Fluency and confidence say nothing about whether an answer is true.

What works, in the order this course builds it:

• Give it the facts. Put the relevant help page in the prompt (RAGRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes., modules 4 and 5), so it answers from text rather than memory.
• Allow it to decline. Tell it to answer only from the pages and to say when they do not cover the question, then check that it does.
• Measure. An eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. with questions the pages do not answer shows how often it guesses anyway (module 7).
• Keep exact facts in code. An order's status comes from your database through a tool, not from the model (module 6).

What you set, and what it costs to get wrong

LessonHow AI writes its answers: one piece at a timeAn AI chat app does not work out a whole answer and then show it. It writes one small piece at a time, and a weighted roll of the die picks each piece. This…

Build it: measure the shop's text in tokens

The support assistant you build in this course answers from the shop's 6 help pages, in a docs/ folder of Markdown files. Before deciding how to use them, measure them. tokens.py does 3 things:

1. counts each help page in tokens and compares it with its word count,
2. prices the simplest design, every page in every prompt, on 2 models, and checks how much of the window it takes,
3. holds a 3-turn conversation and prints the input tokens of each turn.

"""Measure the shop's text in tokens: what each help page costs to send, and how a conversation grows."""
from pathlib import Path

import anthropic

MODEL = "claude-opus-5-5"
PRICES = {"claude-opus-5-5": (4.00, 20.00), "claude-haiku-5-5": (0.10, 0.50)}  # dollars per million tokens

client = anthropic.Anthropic()


def tokens(text: str, model: str = MODEL) -> int:
    """Input tokens for one user message, counted by the API for this model. Free, and no model runs."""
    return client.messages.count_tokens(model=model, messages=[{"role": "user", "content": text}]).input_tokens


# 1. Words are not tokens: count each help page.
pages = sorted(Path("docs").glob("*.md"))
for page in pages:
    text = page.read_text()
    n = tokens(text)
    print(f"{page.name:<14} {len(text.split()):>4} words  {n:>5} tokens  {len(text) / n:.1f} characters a token")

# 2. Put every page in every prompt? Price it before deciding.
everything = "\n\n".join(page.read_text() for page in pages)
email = "My mixer was meant to arrive Monday. Where is it? Order KX-48213."
windows = {model.id: model.max_input_tokens for model in client.models.list()}
print()
for model, (input_price, _) in PRICES.items():
    n = tokens(f"{everything}\n\n<email>\n{email}\n</email>", model)
    per_email = n * input_price / 1_000_000
    print(f"{model:<17} {n:>6} input tokens, {n / windows[model]:.2%} of its window, "
          f"${per_email:.5f} an email, ${per_email * 10_000:.2f} for 10,000 emails, before any output")

# 3. The model remembers nothing between calls: each turn sends the whole conversation again.
conversation = []
print()
for turn, question in enumerate(["Where is order KX-48213?", "Can I change the delivery address?",
                                 "And if it arrives late, can I get a refund?"], start=1):
    conversation.append({"role": "user", "content": question})
    response = client.messages.create(model="claude-haiku-5-5", max_tokens=300, messages=conversation)
    conversation.append({"role": "assistant", "content": response.content})
    print(f"turn {turn}: {response.usage.input_tokens:>4} tokens in, {response.usage.output_tokens:>4} out")

Parts 1 and 2 cost nothing, since counting is free; part 3 makes 3 small calls on Claude Haiku 5.5. The prices are in the script by hand, from Anthropic's pricing page, so check them against the current prices when you run it.

Run it with python tokens.py. This is its output against a stand-in for the API that estimates tokens from characters, so your numbers will differ; the shape will not:

cookware.md     151 words    223 tokens  3.7 characters a token
delivery.md     110 words    170 tokens  3.6 characters a token
knives.md        87 words    131 tokens  3.6 characters a token
mixer.md        119 words    168 tokens  3.6 characters a token
returns.md      100 words    146 tokens  3.6 characters a token
warranty.md      97 words    150 tokens  3.6 characters a token

claude-opus-5-5      956 input tokens, 0.10% of its window, $0.00382 an email, $38.24 for 10,000 emails, before any output
claude-haiku-5-5     956 input tokens, 0.10% of its window, $0.00010 an email, $0.96 for 10,000 emails, before any output

turn 1:   14 tokens in,   15 out
turn 2:   77 tokens in,   15 out
turn 3:  142 tokens in,   15 out

Read it as an engineer would:

• Tokens outnumber words. Every page makes more tokens than words, so a budget planned in words comes out short.
• 6 pages fit easily, and still cost something. They use a tenth of 1% of the window, so for this shop, sending them all is a real option. The same choice on a shop with 600 pages would be 100 times the tokens on every email, and the price difference between models, about 40 times here, would decide the design. Module 4 adds search so the prompt carries only the pages that matter.
• The conversation grows. Each turn's input includes every turn before it. Add a fourth and fifth question and watch the first number climb.

Try 3 changes before moving on. Count the same page with claude-haiku-5-5 and claude-opus-5-5 and compare. Add a system prompt to the counting call and see what it adds. And add the help pages to the first message of the conversation in part 3, then watch every later turn pay for them again; that is the cost module 9's caching removes.