What chunk size for RAG? What the tests show, and how to pick yours

Start with chunks of 200 to 400 tokens, no overlap, cut along the document's own headings and paragraphs, then test 2 or 3 sizes on real questions before you embed everything. That is what the published evidence supports. In Chroma's 2024 test, 200-token chunks found the answer as often as 800-token chunks with 400 tokens of overlap, which is OpenAI's file search default, while sending the model a quarter of the text, and 5 times more of it was the answer.

Chunk size is the setting most RAG systems never question, because a default came with the tool. It decides how often the search finds the answer, how much each question costs, and how much noise the model has to read past.

What a chunk size changes

In retrieval-augmented generationRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes., documents are cut into chunksChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file., each chunk becomes an EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them., and a question pulls back the closest few, often 5, into the prompt. The size of each chunk moves 3 things:

• Whether the answer comes back. Too small and an answer that spans 2 paragraphs is split, and one half may not be retrieved. Too large and one chunk covers several subjects, so its embedding matches each of them weakly.
• How much of the prompt is noise. 5 chunks of 800 tokens is 4,000 tokens per question. If the answer is 2 sentences, nearly all of that is text the model has to read past and you pay for.
• How many vectors you store. Halving the size doubles the chunks, the vectors and the index. Overlap adds more: 400 tokens of overlap on 800-token chunks means every token is embedded twice.

What the tests found

Chroma tested common splitters at several sizes, with OpenAI's text-embedding-3-large, retrieving 5 chunks per question. 2 measures matter: recall, how much of the answer came back, and precision, how much of what came back was the answer.

3 things stand out:

1. Smaller chunks did not lose answers. Recall stayed between 85% and 90% across sizes, and 400 tokens did best of the plain splitters.
2. Precision rose sharply as chunks shrank. At 200 tokens, 5 times as much of the returned text was the answer as at 800.
3. Overlap did not help. 400 tokens with no overlap beat 400 with half overlap on both measures. The default with heavy overlap scored lowest on everything except recall.

Try the settings against your own document count:

Try itChunk size: what a published test measured

Chunk size / overlap (tokens)

800 / 400400 / 200400 / 0200 / 0

Your documents

2000pages

1002,00050,000

Chunks to embed

2,500

Tokens per question

4,000

Answer share

1.5%

Splitter, size / overlap	Recall	Precision
Token splitter, 800 / 400 (like OpenAI's default)	87.9%	1.4%
Recursive, 800 / 400	85.4%	1.5%
Recursive, 400 / 200	88.1%	3.3%
Recursive, 400 / none	89.5%	3.6%
Recursive, 200 / none	88.1%	7%
Cluster semantic, 200best precision	87.3%	8%

Recall bars start at 80%, so small gaps show.

The test has limits. It used particular corpora, one embedding model and 5 chunks per question; precision figures are low everywhere because answers are short and chunks are not. Treat it as a reason to test smaller chunks, not as your answer.

Cut on structure, not on a count

A fixed token count cuts tables in half and glues the end of one section to the start of the next. Most of what overlap is meant to fix comes from that. Better:

• Split on headings first, then paragraphs, then sentences, as a recursive splitter does.
• Keep a table, a list or a code block whole, with the sentence that introduces it.
• Put the document's title and the section heading at the top of every chunk, so a chunk about "the limit" still says which limit.

That last step is a cheap version of Anthropic's contextual retrieval: a model writes 50 to 100 tokens saying where each chunk sits in its document, and that note is embedded with it. In Anthropic's tests it cut failed retrievals by 35%, and by 67% with keyword search and reranking added. Small chunks lose context; this puts it back.

Test your own size in an afternoon

1. Write 30 to 50 questions your users really ask, each with the passage that answers it.
2. Chunk a sample of documents at 200, 400 and 800 tokens.
3. For each size, count how often the answering passage is in the top 5.
4. Pick the smallest size that does not lose answers. It costs least per question and gives the model the least to read past.

Do this before embedding the full set. Every change of size means chunking and embedding everything again. Our vector database size calculator shows what each size does to the number of vectors and the memory they need, the embedding cost calculator what embedding them costs, and the embeddings and search module of our AI engineering course builds a search from scratch. For which model turns chunks into vectors, see how to choose an embedding model. To put chunking, embeddings and search together into a working app, follow how to build a RAG app.