# What is an embedding pipeline?

The pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source.

- Page: https://tecyfy.com/concepts/embedding-pipeline
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: advanced
- Also called: Vector ingestion

A demo of an assistant that answers from documents takes an afternoon: load some PDFs, cut them into chunks, embed them, ask a question. Running one for a company takes a pipeline, because the documents keep changing. A policy is updated, a manual is withdrawn, new tickets arrive. An embedding pipeline keeps the search index in step with the documents it was built from.

## The steps

1. **Find what changed.** New documents, edited ones and deleted ones, ideally without rereading everything. This is [change data capture](https://tecyfy.com/concepts/change-data-capture) applied to documents.
2. **Parse.** Turn each file into clean text, keeping headings and tables. On Databricks, `ai_parse_document` does this for PDFs, Word and PowerPoint files.
3. **Chunk.** Cut the text into passages, the step called [Chunking](https://tecyfy.com/concepts/chunking).
4. **Embed.** Send each chunk to an [Embedding](https://tecyfy.com/concepts/embedding) model, which returns a list of numbers standing for its meaning. OpenAI's `text-embedding-3-small`, for example, returns 1,536 numbers per chunk by default.
5. **Write to the index.** Store each vector with its text and a pointer to its source, in a [vector database](https://tecyfy.com/concepts/vector-database) or search index.
6. **Remove what was deleted.** The step most demos skip.

## Keeping in step

On Databricks the pattern is built in. The parsed and chunked text lives in a Delta table with its change data feed turned on, so every insert, update and delete is recorded. An AI Search **Delta Sync** index reads that feed and updates itself: new chunks are added, changed ones replaced, deleted ones removed. It can also compute the embeddings itself, with a model you name.

The general lesson holds everywhere. The index is a copy of the documents, and every copy drifts unless something keeps it in step. An assistant quoting a withdrawn policy is almost never a model problem. It is a delete that never reached the index.

(An interactive demo goes here on the page: https://tecyfy.com/concepts/embedding-pipeline)

## Choose the model like a database, not a setting

Every vector in an index comes from one embedding model, and vectors from 2 models cannot be compared. So changing the model means embedding every chunk again into a new index. On Databricks a self-managed index cannot be switched over; you build a new one.

That makes 2 decisions worth getting right early. Keep the parsed text, not only the vectors, so a re-embed is a rerun rather than a rebuild from the sources. And treat the model name as part of the index, recorded with it, so nobody mixes vectors by accident. What a full re-embed costs on each model is in the [embedding cost tool](/prices/embedding-cost).

## Better than vectors alone

Vector search finds passages that mean the same thing as the question. It is weaker on exact words: a product code, an error number, a person's name. Hybrid search adds keyword scoring alongside the vectors, and Databricks AI Search offers it with BM25, the standard keyword method. Anthropic's tests on contextual retrieval found the same: adding keyword search to embeddings cut failed retrievals further than embeddings alone.

## Why it matters

For an assistant built on company knowledge, the embedding pipeline is the [data pipeline](https://tecyfy.com/concepts/data-pipeline) that matters most, and the one most often built as a one-off script. It deserves the same treatment as any other: [Orchestration](https://tecyfy.com/concepts/orchestration), [quality checks](https://tecyfy.com/concepts/data-quality) on the parsed text, alerts when it fails, and [lineage](https://tecyfy.com/concepts/data-lineage) from each chunk back to the document it came from.

## Related

- [Chunking](https://tecyfy.com/concepts/chunking): Cutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file.
- [Embedding](https://tecyfy.com/concepts/embedding): A list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them.
- [Vector Database](https://tecyfy.com/concepts/vector-database): A database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning.
- [Change Data Capture](https://tecyfy.com/concepts/change-data-capture): Copying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again.

## Sources

- [Databricks AI Search](https://docs.databricks.com/aws/en/vector-search/vector-search)
- [Use Delta Lake change data feed](https://docs.databricks.com/aws/en/delta/delta-change-data-feed)
- [Vector embeddings](https://developers.openai.com/api/docs/guides/embeddings)
- [Transform unstructured data using AI Functions](https://docs.databricks.com/aws/en/large-language-models/ai-functions)
- [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)