What is a tokenizer?

Before a model can read your message, something has to cut it into tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.. That something is the tokenizer. It runs before the model, on the way in, and again after it, turning the tokens the model writes back into text you can read.

Why not whole words, or single letters?

Both have a problem. English alone has over 500,000 words, far too many for a model to keep one entry for each, and a new word or a typo would have no entry at all. Single letters avoid that, but make every text many times longer, and length costs time and money.

So most tokenizers sit in between. They keep common words whole and split rarer ones into pieces that still carry meaning. Hugging Face's example: "annoyingly" becomes "annoying" and "ly". A word the tokenizer has never seen can still be built from smaller pieces it knows.

Every model family has its own

The tokenizer is fixed when the model is trained, and different makers build different ones. That is why the same sentence gives one count on GPT and another on Claude or Gemini. OpenAI publishes its tokenizer as a library called tiktoken, so you can count GPT tokens yourself before you send anything.

A new tokenizer can change your bill without any change in price. Anthropic's tokenizer for its newest models cuts the same text into roughly 30% more tokens than the one before. At the same price per token, the same prompt then costs more.

Your language changes the count

The language matters as well. OpenAI notes that the same text can give a different number of tokens depending on the model, its encoding and the language it is written in. If you write prompts in Hindi, Japanese or Polish, test the count in that language before you trust a rule of thumb made for English.

What to do with this

When you compare models on cost, run the same real text through each and compare the token counts, not just the prices. And when you move to a newer model of the same family, count again: the tokenizer may have changed with it.