How to prepare company data for RAG: clean, chunk, tag

To prepare company data for RAG, treat it as a data pipeline with 6 steps: collect the questions people will ask, extract and clean the text, split it at the document's own structure, tag every chunk with its source, version and who may read it, add a line of context, then load it into an index that updates when the documents change. Most wrong answers from a RAG system start in one of those steps, not in the model: a stale policy still in the index, a heading cut off from its paragraph, a manager-only page any employee can retrieve.

Here is what each step changes, on a small made-up HR policy with an older copy still in the folder:

Try itPrepare a policy for RAG, step by step

Steps

·Raw text·Remove headers and contents·Split at headings·Add context·Add metadata and filter

EmployeeManager

• Fix4 chunks carry page headers or contents lines, which get embedded as if they were content.
• FixThe chunk with the answer (16 weeks) lost its heading, so nothing in it says it is about parental leave.
• FixThe 2024 copy, which says 12 weeks, can still be retrieved and quoted.
• FixManager-only guidance can be retrieved for an employee.

1. chunk 1

   NORTHWIND LTD · INTERNAL · PAGE 1 OF 2 Contents Parental leave .......... 1 Manager guidance .......... 2 Parental leave Employees with

2. chunk 2

   26 weeks of service get 16 weeks of parental leave at full pay, then 10 weeks at half pay. Tell your manager

3. chunk 3

   at least 8 weeks before your leave starts. You can split the leave into 2 blocks. NORTHWIND LTD · INTERNAL · PAGE

4. chunk 4

   2 OF 2 Manager guidance Approve a request within 5 working days. Cover is paid from the team budget, not from the

5. chunk 5

   leave budget.

6. chunk 6

   NORTHWIND LTD · INTERNAL · PAGE 1 OF 1 Parental leave Employees get 12 weeks of parental leave at full pay.

With no steps on, the reader gets every failure at once. Turn on the steps one by one and watch which ones remove which failure. The last one to fix is the one people miss: with fixed-size chunks and a permission filter, a chunk that mixes employee text with manager text gets hidden from employees entirely, so they lose part of their own policy. Splitting at headings first is what makes the filter clean.

Start from the questions

Before touching a file, write down questions real people will ask, and for each one, the passage that answers it. Microsoft's RAG guide puts this in its first phase, done in parallel with collecting the documents, because the 2 depend on each other: the questions must be answerable from the content, and a question no document answers shows a gap before you build anything.

The same list becomes your test set. Every change later, a cleaning rule, a chunk size, an embedding model, gets judged by whether the right passage still comes back for those questions. Without it you are tuning by feel. Our concept page on Retrieval EvaluationRetrieval EvaluationTesting the search step of a RAG system on its own, with questions whose answering passages you know, and scoring it with recall at k, the share of those passages found in the top k results. covers how to score it.

While you collect, sort the documents by kind (contracts, product sheets, tickets) and by format (PDF, Word, HTML, scans). Each pair usually needs its own extraction and its own test examples; Microsoft suggests at least 2 per variant.

Clean out what carries no meaning

Text pulled from a PDF arrives with things a reader skips without noticing: page headers and footers, "Page 3 of 12", a contents list with dotted lines, copyright lines, watermarks. Embedded, they are content like any other. A contents list is a trap of its own: it is made of your section titles, so it can look relevant to a question about any section.

Clean in a separate column and keep the original. Microsoft's guide stores the cleaned text for embedding next to the untouched chunk, which is what the reader sees and what an answer cites. And test before cleaning harder: removing stop words shrinks the text, but "not" is a stop word, and "leave is not paid" means the opposite of "leave is paid".

2 things belong in the cleaning step that are not about formatting:

• Replaced versions. If the 2024 policy and the 2026 policy are both in the folder, both get indexed, and the search has no way to know which is current. OWASP lists conflicting sources as a risk for RAG systems. Drop superseded files, or at least mark them so they can be filtered out.
• Instructions inside documents. Anyone who can edit a wiki page that gets indexed can write text aimed at the model. Microsoft's guidance is never to treat document content as instructions, and to flag content that contains them.

Split at the document's own structure

Cut at headings, so each chunk is one section with its title. A fixed word count is simpler but cuts wherever the count runs out: a heading ends one chunk while its paragraph starts the next, and one chunk can end up holding the last line of one section and the first of another. A long table may need its header row repeated in each chunk, or the numbers lose their columns.

How large each piece should be is its own question, with published tests behind it, in our guide to RAG chunk size. Our data engineering course has a module that splits Markdown at headings into a Delta table with PySpark, step by step.

Tag every chunk

Metadata is what lets the search filter, and what lets an answer cite. It sits in the same row as the embedding. The fields that matter in nearly every company:

Microsoft's enrichment guide adds a title, a summary and the questions the chunk can answer, each written by a model, which help when questions are worded far from the document's own words.

The permission field deserves the most care. A RAG system that indexes HR, finance and engineering documents into one store and searches them all for every user has handed everyone every document. OWASP's guidance is permission-aware stores that partition data by user group. In practice: copy each document's access list onto its chunks when you create them, and filter on it at query time for the person asking. Note what Microsoft says about its own version of this pattern: the identity stored on a document is just a string in a filter. It protects nothing unless your application passes the real identity of the person asking, and unless the list on the chunk stays in step with the list on the source.

Add a line of context

A chunk that says "Revenue grew by 3% over the previous quarter" does not say which company or which quarter. Anthropic's contextual retrieval fixes that by asking a model, for each chunk, for a short context that places it in its document, usually 50 to 100 tokens, and putting that context in front of the chunk before embedding it and before building the keyword index.

In Anthropic's tests across several knowledge domains, that alone cut the share of questions whose right passage was missing from the top 20 results by 35%. Adding keyword search (Hybrid SearchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found.) on the same context took the cut to 49%, and RerankingRerankingA second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order. the results to 67%. With prompt caching, so the full document is sent once rather than once per chunk, Anthropic put the one-time cost at $1.02 per million document tokens for its example sizes.

A cheaper start: put the document title and section path in front of every chunk with no model at all. It does not capture what the model would, but it fixes the case in the widget above: a chunk that never says what it is about.

Keep it as tables, and keep it current

A RAG index is a derived table like any other, and it goes stale the same way. Build it as one:

CREATE TABLE rag.chunks (
  chunk_id      STRING,         -- sha2 of document id, section and text
  document_id   STRING,
  version       STRING,
  is_current    BOOLEAN,
  section       STRING,
  readers       ARRAY<STRING>,  -- groups copied from the source's access list
  context       STRING,
  chunk_text    STRING,         -- what readers see and answers cite
  embed_text    STRING,         -- cleaned, with the context in front
  updated_at    TIMESTAMP
)
TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');

With a content hash as the ID, a nightly run only embeds chunks whose hash is new, and deletes chunks whose document disappeared or whose is_current flag turned false. Then let the index follow the table instead of loading it by hand. On Databricks, an AI Search Delta Sync index does that: it syncs from a Delta table with change data feed on when the table is updated. The Embedding PipelineEmbedding PipelineThe pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source. concept page draws the whole flow. If you are building the application on top, our guide to building a RAG app picks up where this table ends.

Then rerun the questions from the first step. If the right passage comes back for each of them, for the right reader, from the current version, the data is ready.