What is a token in AI?

A language model does not read letters or whole words. It reads tokens: short pieces of text that can be a single character, part of a word, a whole word or a punctuation mark. Common words are often 1 token each, and rarer words are split into several. Spaces count too, because they change where the text is cut.

As a rough guide for English, 1 token is about 4 characters, or three-quarters of a word, in OpenAI's counting. For Claude it is about 3.5 characters. The language changes the count too, and the same sentence gives different counts on different models, because each maker cuts text with its own TokenizerTokenizerThe part that cuts text into the tokens a model reads, and turns the model's tokens back into text. Each model family has its own..

Why it matters to you

Almost everything you pay for or run into is counted in tokens.

• The bill. API prices are per million input tokens and per million output tokens, and output costs more.
• The limit. The context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out., the most a model can see at once, is a number of tokens.
• The speed. A model writes 1 token at a time, each one predicted from everything before it. A longer answer takes longer.

Not only text

Pictures and sound are turned into tokens as well. For Gemini, an image up to 384 pixels on each side counts as 258 tokens, and audio counts as 32 tokens for every second. A long recording can fill a window faster than you would guess.

The tokens you never see

A model that thinks before it answers writes its reasoning in tokens too. You may not see them, but they are counted and billed as output, at the higher price. So 2 models with the same price per token can give very different bills for the same job: the one that thinks longer, or takes more steps, uses more tokens.

How to count them

OpenAI publishes its tokenizer so you can count GPT tokens yourself, and the other makers' API documentation explains how to count theirs. When cost matters, count a real example rather than guess from the word count. And compare the cost of the whole job, not the price per token.