The full answer
Evaluate the two halves separately and then end to end. Retrieval: did the needed passages come back, and near the top (recall@k, context recall and precision)? Generation: does the answer stick to those passages (faithfulness) and actually answer the question (relevance)? End to end: is it correct against a reference answer? Separate scores tell you where to fix, not only that something is wrong.
The test set
Questions from real users, each with:
- the reference answer (or the key facts it must contain),
- the ids of the passages that support it,
- some questions with no answer in the corpus, where the right response is "I don't know".
Retrieval metrics
- Recall@k: the share of questions where a supporting passage is in the top k. This is the single most useful number.
- Ranking quality (MRR, nDCG): how high the right passage ranks.
- Ragas calls the LLM-graded versions context recall (does the retrieved context contain what the reference answer needs?) and context precision (are relevant chunks ranked above irrelevant ones?).
Generation metrics
- Faithfulness: the share of claims in the answer that the retrieved context supports. Ragas computes it as supported claims divided by total claims, from 0 to 1. Low faithfulness means the model is adding things.
- Answer relevance: does it answer the question asked, rather than something nearby?
- Citation accuracy: do the cited passages really support the sentences they are attached to?
End-to-end metrics
- Correctness against the reference: in code for short factual answers, with an LLM judge and rubric for longer ones.
- Abstention: on unanswerable questions, does it say it doesn't know?
- Latency and cost per question.
Reading the results together
| Retrieval | Faithfulness | Correctness | Diagnosis |
|---|---|---|---|
| Low | any | Low | Fix parsing, chunking or search first |
| High | Low | Low | Fix the prompt or model: it ignores or adds to the sources |
| High | High | Low | Sources are wrong or out of date, or the reference is wrong |
| High | High | High | Good: watch for regressions |
In production
- Log retrieved chunk ids with every answer.
- Sample live answers for judge and human review.
- Collect user feedback (thumbs, "this didn't help") and turn bad cases into tests.
- Rerun the full set when you change chunking, embeddings, the reranker, the prompt or the model.
A strong close: "LLM-graded metrics are only as good as the judge, so I spot-check them against human labels before trusting a change in score."
Last updated 16 October 2026
Sources