How do you chunk documents for retrieval, and how do you know the chunking is good?

Split documents along their own structure (sections, headings, paragraphs) into pieces that each make sense alone, typically a few hundred tokens, with a little overlap. Keep the document's title and section with every chunk. Then judge the chunking by retrieval results on real questions, not by eye: the right chunk size is whichever finds the needed passage most often.

Why chunking matters

Retrieval returns chunks, not documents. A chunk that is too small loses its meaning ("It must be renewed every 12 months" — what must?). A chunk that is too large mixes several topics, so its embedding matches none of them well, and it crowds the prompt. Most RAG systems that "can't find" an answer are failing here.

A sound default (chunking)

1. Parse properly first. Extract text from PDFs and HTML with headings, lists and tables intact. Bad extraction makes every later step worse.
2. Split on structure. Sections, then paragraphs, then sentences. Never cut a table row or a code block in half.
3. Size and overlap. Start at roughly 300 to 800 tokens with 10 to 20% overlap, then tune on your questions.
4. Carry context. Store the document title, section heading, date and source with each chunk. Prefix the heading to the text you embed.

Contextual chunks

Anthropic's contextual retrieval goes one step further. A model writes a short explanation (50 to 100 tokens) of where each chunk sits in its document ("This chunk is from ACME's Q2 2025 report and covers revenue growth…"), and that text is prepended before embedding and keyword indexing. Combined with keyword search, it cut failed retrievals by 49% in their tests, and by 67% with reranking added.

Special content

• Tables: keep a table whole, or one row per chunk with its column headers repeated.
• Code: split by function or class.
• FAQs, tickets: one question and its answer per chunk.
• Very short documents: do not split them at all.

How to know it is good

• Build 50 to 100 real questions, each with the passage that answers it.
• Measure recall@k: how often the right passage is in the top k results. Compare chunk sizes and strategies on that number.
• Ragas' context recall metric measures the same thing against reference answers.
• Read failures. If the right passage exists but is split across 2 chunks, adjust the splitter. If it is found but ranked 15th, add reranking.

Common mistakes: one fixed size for every document type, losing headings, and tuning chunk size by looking at outputs rather than measuring retrieval.