What is an embedding pipeline?

A demo of an assistant that answers from documents takes an afternoon: load some PDFs, cut them into chunks, embed them, ask a question. Running one for a company takes a pipeline, because the documents keep changing. A policy is updated, a manual is withdrawn, new tickets arrive. An embedding pipeline keeps the search index in step with the documents it was built from.

The steps

1. Find what changed. New documents, edited ones and deleted ones, ideally without rereading everything. This is change data captureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again. applied to documents.
2. Parse. Turn each file into clean text, keeping headings and tables. On Databricks, ai_parse_document does this for PDFs, Word and PowerPoint files.
3. Chunk. Cut the text into passages, the step called ChunkingChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file..
4. Embed. Send each chunk to an EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them. model, which returns a list of numbers standing for its meaning. OpenAI's text-embedding-3-small, for example, returns 1,536 numbers per chunk by default.
5. Write to the index. Store each vector with its text and a pointer to its source, in a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning. or search index.
6. Remove what was deleted. The step most demos skip.

Keeping in step

On Databricks the pattern is built in. The parsed and chunked text lives in a Delta table with its change data feed turned on, so every insert, update and delete is recorded. An AI Search Delta Sync index reads that feed and updates itself: new chunks are added, changed ones replaced, deleted ones removed. It can also compute the embeddings itself, with a model you name.

The general lesson holds everywhere. The index is a copy of the documents, and every copy drifts unless something keeps it in step. An assistant quoting a withdrawn policy is almost never a model problem. It is a delete that never reached the index.

Try itSearch an index by meaning

The question

·What can I spend on a London hotel?·How much holiday do I get?·I lost my laptop. What now?

Deletes reach the index

NoYes

Withdraw the 2024 policy

London hotel £180Other hotels £120Book 14 days aheadLondon hotel £150Business class25 days holidayCarry over 5 daysParental leaveReset passwordLost laptop

The questionReturnedNot returnedFrom the withdrawn policy

What the search returns

1. 1A hotel night in London may cost up to £180.
   Travel policy 2026
2. 2Hotels in London: up to £150 a night.
   Travel policy 2024
3. 3Book travel through the portal 14 days ahead.
   Travel policy 2026

Choose the model like a database, not a setting

Every vector in an index comes from one embedding model, and vectors from 2 models cannot be compared. So changing the model means embedding every chunk again into a new index. On Databricks a self-managed index cannot be switched over; you build a new one.

That makes 2 decisions worth getting right early. Keep the parsed text, not only the vectors, so a re-embed is a rerun rather than a rebuild from the sources. And treat the model name as part of the index, recorded with it, so nobody mixes vectors by accident. What a full re-embed costs on each model is in the embedding cost tool.

Better than vectors alone

Vector search finds passages that mean the same thing as the question. It is weaker on exact words: a product code, an error number, a person's name. Hybrid search adds keyword scoring alongside the vectors, and Databricks AI Search offers it with BM25, the standard keyword method. Anthropic's tests on contextual retrieval found the same: adding keyword search to embeddings cut failed retrievals further than embeddings alone.

Why it matters

For an assistant built on company knowledge, the embedding pipeline is the data pipelineData PipelineA set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives. that matters most, and the one most often built as a one-off script. It deserves the same treatment as any other: OrchestrationOrchestrationRunning the steps of data pipelines in the right order and at the right time, retrying failures and alerting someone when a step breaks., quality checksData QualityHow far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail. on the parsed text, alerts when it fails, and lineageData LineageThe record of where a piece of data came from and everything it passed through, from source tables to the dashboards, models and AI answers built on it. from each chunk back to the document it came from.