# How do you measure whether a search or RAG system found the right passages? Testing the search step of a RAG system on its own, with questions whose answering passages you know, and scoring it with recall at k, the share of those passages found in the top k results. - Page: https://tecyfy.com/concepts/retrieval-evaluation - Subject: Data engineering in the AI era (https://tecyfy.com/concepts) - Level: intermediate - Also called: Recall at k, Recall@k, Precision at k, Mean Reciprocal Rank, MRR, RAG Evaluation When a RAG app gives a wrong answer, there are 2 places it can have gone wrong. The search may have missed the passage that held the answer, or the model may have had it and still answered badly. A score for the whole app cannot tell them apart. So you test the search on its own, before the model sees anything. ## What you need A list of questions, and for each one the passage or passages that answer it. "How long until I get my money back?" is answered by the section "When your refund arrives". Write the list from real questions where you can: the support inbox, the search box's logs, the failures you found when reading your app's [traces](https://tecyfy.com/concepts/tracing). A model can draft extra questions from your documents to fill gaps, but a question written while looking at the answer tends to reuse its words, which is easier for a search than the words customers use. 20 to 50 questions, as for any [eval set](https://tecyfy.com/concepts/eval-set), is a start. Store each passage's id with the question, so the test is code and runs in a second. ## The numbers **Recall at k** is the main one. Run the search, take its top k results, and count the share of the right passages among them. If a question has 1 right passage and it is in the top 5, recall@5 is 100% for that question. Average over all questions. It answers the question that matters for RAG: did the model get the text it needed? Choose k to match what you send the model. If the prompt gets the top 5 passages, measure recall@5. Anthropic measured its retrieval tests as 1 minus recall@20, the share of right passages missing from the top 20, and found that 20 passages did better than 10 or 5. 2 others help: - **Precision at k**, the share of the top k that are right. Low precision means the prompt is full of passages that do not help, which costs [tokens](https://tecyfy.com/concepts/token) and can distract the model. - **Mean reciprocal rank (MRR)**, the average of 1 divided by the position of the first right passage. First place scores 1, third place 1/3. It shows whether the best passage is near the top, which matters when k is small. ## Use it to decide Retrieval has many settings: chunk size, the embedding model, keyword search beside vector search ([hybrid search](https://tecyfy.com/concepts/hybrid-search)), a [Reranking](https://tecyfy.com/concepts/reranking) step, k itself. Each change is a guess until recall moves. Score every variant on the same questions, read the questions it missed, and keep the change only if it helps on the questions your users ask. A good recall score is not the end. Then test the answers: that each one sticks to the passages it was given, which is [Grounding](https://tecyfy.com/concepts/grounding), and that it cites them. Those are graded like any other eval, with code where a rule can decide and a model judge where it cannot. ## Related - [Eval Set](https://tecyfy.com/concepts/eval-set): The list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. - [RAG](https://tecyfy.com/concepts/rag): A way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes. - [Hybrid Search](https://tecyfy.com/concepts/hybrid-search): Running a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found. - [Reranking](https://tecyfy.com/concepts/reranking): A second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order. - [Grounding](https://tecyfy.com/concepts/grounding): Making a model answer from sources you give it, such as retrieved documents, and only from them, so every statement can be traced to a passage and a missing answer is admitted, not invented. ## Sources - [Evaluation of ranked retrieval results](https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html) - [Mean reciprocal rank](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) - [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) - [AI evals: everything you need to know](https://hamel.dev/blog/posts/evals-faq/)