How do you measure whether a search or RAG system found the right passages?

When a RAG app gives a wrong answer, there are 2 places it can have gone wrong. The search may have missed the passage that held the answer, or the model may have had it and still answered badly. A score for the whole app cannot tell them apart. So you test the search on its own, before the model sees anything.

What you need

A list of questions, and for each one the passage or passages that answer it. "How long until I get my money back?" is answered by the section "When your refund arrives". Write the list from real questions where you can: the support inbox, the search box's logs, the failures you found when reading your app's tracesTracingRecording every step an LLM app or agent takes for one request, each model call, tool call and retrieval with its input, output, tokens and time, so you can see why an answer went wrong.. A model can draft extra questions from your documents to fill gaps, but a question written while looking at the answer tends to reuse its words, which is easier for a search than the words customers use.

20 to 50 questions, as for any eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped., is a start. Store each passage's id with the question, so the test is code and runs in a second.

The numbers

Recall at k is the main one. Run the search, take its top k results, and count the share of the right passages among them. If a question has 1 right passage and it is in the top 5, recall@5 is 100% for that question. Average over all questions. It answers the question that matters for RAG: did the model get the text it needed?

Choose k to match what you send the model. If the prompt gets the top 5 passages, measure recall@5. Anthropic measured its retrieval tests as 1 minus recall@20, the share of right passages missing from the top 20, and found that 20 passages did better than 10 or 5.

2 others help:

• Precision at k, the share of the top k that are right. Low precision means the prompt is full of passages that do not help, which costs tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. and can distract the model.
• Mean reciprocal rank (MRR), the average of 1 divided by the position of the first right passage. First place scores 1, third place 1/3. It shows whether the best passage is near the top, which matters when k is small.

Use it to decide

Retrieval has many settings: chunk size, the embedding model, keyword search beside vector search (hybrid searchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found.), a RerankingRerankingA second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order. step, k itself. Each change is a guess until recall moves. Score every variant on the same questions, read the questions it missed, and keep the change only if it helps on the questions your users ask.

A good recall score is not the end. Then test the answers: that each one sticks to the passages it was given, which is GroundingGroundingMaking a model answer from sources you give it, such as retrieved documents, and only from them, so every statement can be traced to a passage and a missing answer is admitted, not invented., and that it cites them. Those are graded like any other eval, with code where a rule can decide and a model judge where it cannot.