# What is a tokenizer?

The part that cuts text into the tokens a model reads, and turns the model's tokens back into text. Each model family has its own.

- Page: https://tecyfy.com/concepts/tokenizer
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Tokeniser
- The lesson: How AI writes its answers: one piece at a time (https://tecyfy.com/learn/how-ai-works/how-ai-writes)
- Last checked: 2026-10-14

Before a model can read your message, something has to cut it into [tokens](https://tecyfy.com/concepts/token). That something is the tokenizer. It runs before the model, on the way in, and again after it, turning the tokens the model writes back into text you can read.

## Why not whole words, or single letters?

Both have a problem. English alone has over 500,000 words, far too many for a model to keep one entry for each, and a new word or a typo would have no entry at all. Single letters avoid that, but make every text many times longer, and length costs time and money.

So most tokenizers sit in between. They keep common words whole and split rarer ones into pieces that still carry meaning. Hugging Face's example: "annoyingly" becomes "annoying" and "ly". A word the tokenizer has never seen can still be built from smaller pieces it knows.

## Every model family has its own

The tokenizer is fixed when the model is trained, and different makers build different ones. That is why the same sentence gives one count on GPT and another on Claude or Gemini. OpenAI publishes its tokenizer as a library called tiktoken, so you can count GPT tokens yourself before you send anything.

A new tokenizer can change your bill without any change in price. Anthropic's tokenizer for its newest models cuts the same text into roughly 30% more tokens than the one before. At the same price per token, the same prompt then costs more.

## Your language changes the count

The language matters as well. OpenAI notes that the same text can give a different number of tokens depending on the model, its encoding and the language it is written in. If you write prompts in Hindi, Japanese or Polish, test the count in that language before you trust a rule of thumb made for English.

## What to do with this

When you compare models on cost, run the same real text through each and compare the token counts, not just the prices. And when you move to a newer model of the same family, count again: the tokenizer may have changed with it.

## Related

- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.
- [Byte Pair Encoding](https://tecyfy.com/concepts#byte-pair-encoding): A sub‑word tokenization algorithm based on merging frequent symbol pairs
- [Subword Token](https://tecyfy.com/concepts#subword-token): A token representing a fragment of a word, enabling open‑vocabulary modeling.

## Sources

- [LLM Course, chapter 2: Tokenizers](https://huggingface.co/learn/llm-course/chapter2/4), Hugging Face, 2026-10-14. English has over 500,000 words, which is too many for a tokenizer that works word by word. Splitting into single characters makes the text far longer. Subword tokenizers keep frequent words whole and split rare words into parts that still carry meaning, such as "annoyingly" into "annoying" and "ly".
- [tiktoken](https://github.com/openai/tiktoken), OpenAI, GitHub, 2026-10-14. OpenAI publishes its tokenizer, and the library's model list maps the GPT-5 models to the o200k_base encoding.
- [Understanding and counting tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them), OpenAI Help Center, 2026-10-14. The same text can give different token counts depending on the model, its encoding and the language.
- [Claude Platform release notes](https://platform.claude.com/docs/en/release-notes/api), Anthropic, 2026-10-14. Claude Fable 5 and Claude Mythos 5 use the tokenizer introduced with Claude Opus 4.7, and the same text produces roughly 30% more tokens. Claude Sonnet 5's new tokenizer also produces about 30% more tokens.