# What is chunking, and how big should a chunk be? Cutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file. - Page: https://tecyfy.com/concepts/chunking - Subject: Data engineering in the AI era (https://tecyfy.com/concepts) - Level: intermediate - Also called: Text splitting An assistant that answers from your documents does not read all of them for every question. It searches them first, and passes the few passages that look relevant into the prompt. That is [retrieval-augmented generation](https://tecyfy.com/concepts/rag). Chunking is the step before the search: cutting each document into the passages the search can return. ## Why not embed the whole document? Search over documents usually works by [Embedding](https://tecyfy.com/concepts/embedding): each piece of text becomes a list of numbers, and pieces close in meaning get close numbers. One list of numbers for a 40-page handbook has to stand for holidays, expenses, security and the dress code all at once. It matches every question a little and none of them well. Cut the handbook into passages and each one has a single subject, so a question about expenses finds the expenses passage. It also keeps the prompt small: the model gets the 3 paragraphs it needs, not 40 pages. ## How big a chunk? There is no single right size, but defaults show the usual range. OpenAI's file search splits files into chunks of 800 [tokens](https://tecyfy.com/concepts/token) with 400 tokens of overlap by default, and allows anything from 100 to 4,096. The trade-off is plain. Small chunks match precisely but may cut an answer in half. Large chunks keep context but blur the match and fill the prompt. **Overlap**, repeating the end of one chunk at the start of the next, means a sentence cut at a boundary still appears whole in one of them. (An interactive demo goes here on the page: https://tecyfy.com/concepts/chunking) ## Cut along the document's own lines A fixed count of tokens ignores structure. It will cut a table in half and join the last line of one section to the first of the next. Better chunking follows the document: split on headings, keep a table whole, keep a list with the sentence that introduces it. That needs the document parsed first, so the pipeline knows where the headings and tables are. On Databricks, `ai_parse_document` pulls text, tables and layout from PDFs, Word and PowerPoint files, and `ai_prep_search`, in Beta, turns the result into search-ready chunks. ## The chunk that forgot where it came from A chunk taken out of its document can lose what it is about. A passage saying sales rose 3% on the quarter before is no help to a search if it never names the company or the quarter. One fix is to add that context back. Anthropic calls its method **contextual retrieval**: before embedding, a model writes a short note for each chunk, usually 50 to 100 tokens, saying where it sits in the document, and the note is stored with the chunk. In Anthropic's own tests this cut failed retrievals by 35%, and by 67% combined with keyword search and a reranking step. A cheaper step in the same spirit is to put the document's title and section heading at the top of every chunk. ## Changing your mind is expensive Every chunk is embedded, and the vectors are stored in an index. Change the chunk size and every document must be chunked and embedded again. So choose with a test: a set of real questions and the passages that answer them, checked against each setting before you commit. An [embedding pipeline](https://tecyfy.com/concepts/embedding-pipeline) that keeps the parsed text makes that rerun cheap, and the [embedding cost tool](/prices/embedding-cost) shows what each setting costs on every model. ## Related - [Embedding Pipeline](https://tecyfy.com/concepts/embedding-pipeline): The pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source. - [RAG](https://tecyfy.com/concepts/rag): A way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes. - [Embedding](https://tecyfy.com/concepts/embedding): A list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them. ## Sources - [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) - [Retrieval](https://developers.openai.com/api/docs/guides/retrieval) - [Transform unstructured data using AI Functions](https://docs.databricks.com/aws/en/large-language-models/ai-functions) - [Databricks AI Search](https://docs.databricks.com/aws/en/vector-search/vector-search)