What is chunking, and how big should a chunk be?
Cutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file.
intermediate · Also called Text splitting
An assistant that answers from your documents does not read all of them for every question. It searches them first, and passes the few passages that look relevant into the prompt. That is retrieval-augmented generationRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes.. Chunking is the step before the search: cutting each document into the passages the search can return.
Why not embed the whole document?
Search over documents usually works by EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them.: each piece of text becomes a list of numbers, and pieces close in meaning get close numbers. One list of numbers for a 40-page handbook has to stand for holidays, expenses, security and the dress code all at once. It matches every question a little and none of them well.
Cut the handbook into passages and each one has a single subject, so a question about expenses finds the expenses passage. It also keeps the prompt small: the model gets the 3 paragraphs it needs, not 40 pages.
How big a chunk?
There is no single right size, but defaults show the usual range. OpenAI's file search splits files into chunks of 800 tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. with 400 tokens of overlap by default, and allows anything from 100 to 4,096.
The trade-off is plain. Small chunks match precisely but may cut an answer in half. Large chunks keep context but blur the match and fill the prompt. Overlap, repeating the end of one chunk at the start of the next, means a sentence cut at a boundary still appears whole in one of them.
The policy, 148 words
Open a chunk
Chunk 3 · words 51 to 75 · part of the answer
business class needs a director to approve it before booking. Hotels should be close to where you are working. A hotel night in London may
The answer is cut across chunks 3 and 4, and neither holds all of it. A search may return one half, and the model never sees £180 next to London.
A real pipeline counts tokens rather than words, and the better ones cut at sentences and headings. This one cuts blindly at a fixed count, which is where the trouble starts.
Cut along the document's own lines
A fixed count of tokens ignores structure. It will cut a table in half and join the last line of one section to the first of the next. Better chunking follows the document: split on headings, keep a table whole, keep a list with the sentence that introduces it.
That needs the document parsed first, so the pipeline knows where the headings and tables are. On Databricks, ai_parse_document pulls text, tables and layout from PDFs, Word and PowerPoint files, and ai_prep_search, in Beta, turns the result into search-ready chunks.
The chunk that forgot where it came from
A chunk taken out of its document can lose what it is about. A passage saying sales rose 3% on the quarter before is no help to a search if it never names the company or the quarter.
One fix is to add that context back. Anthropic calls its method contextual retrieval: before embedding, a model writes a short note for each chunk, usually 50 to 100 tokens, saying where it sits in the document, and the note is stored with the chunk. In Anthropic's own tests this cut failed retrievals by 35%, and by 67% combined with keyword search and a reranking step. A cheaper step in the same spirit is to put the document's title and section heading at the top of every chunk.
Changing your mind is expensive
Every chunk is embedded, and the vectors are stored in an index. Change the chunk size and every document must be chunked and embedded again. So choose with a test: a set of real questions and the passages that answer them, checked against each setting before you commit. An embedding pipelineEmbedding PipelineThe pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source. that keeps the parsed text makes that rerun cheap, and the embedding cost tool shows what each setting costs on every model.
Sources