The full answer
Split the pipeline and test each half on its own. For each wrong answer, check first whether the right passage was retrieved at all. If not, it is a retrieval problem (parsing, chunking, search, filters). If it was retrieved and the answer still contradicts it or adds claims, it is a generation problem (prompt, context order, model). Logging the retrieved chunks with every answer makes this a 5-minute check instead of a guess.
Step 1: log everything
For each question, store the query sent to search (after any rewriting), the retrieved chunk ids with scores, the final prompt and the answer. Without this you cannot tell the halves apart.
Step 2: sort the failures
Take 30 to 50 wrong answers and label each:
| What you see | Fault | Typical fixes |
|---|---|---|
| The answer is not in the corpus | Content | Add the document, or make the system say it does not know |
| It is in the corpus but not retrieved | Retrieval | Better parsing and chunking, hybrid search, query rewriting, check filters |
| It was retrieved but ranked low and cut off | Ranking | Reranking, pass more chunks |
| It was retrieved, but the answer ignores or contradicts it | Generation | Prompt: "answer only from the sources; say if they don't contain it"; quotes and citations |
| The sources disagree, or are out of date | Data | Dates in metadata, prefer the newest, remove superseded documents |
Step 3: measure each half
- Retrieval: recall@k against labelled passages. Ragas calls the related metrics context recall (was the needed information retrieved?) and context precision (were the relevant chunks ranked high?).
- Generation: faithfulness, the share of the answer's claims that the retrieved context supports. A low score with good recall means the model is adding things.
- End to end: correctness against a reference answer.
Things that look like generation faults but are not
- Position. "Lost in the Middle" (Liu et al.) found models use information at the start or end of a long context better than in the middle. Put the best chunks first, and do not pad the prompt with weak ones.
- Query mismatch. Users ask "can I get my money back?" while documents say "refund policy". Query rewriting or hybrid search fixes this. It is retrieval, not the model.
- Access filters. The right document exists, but a permissions filter removed it. That is correct behaviour, so the answer should say so.
Step 4: lock it in
Add every fixed failure to the evaluation set, and run it on each change of prompt, chunking or model, so the fault cannot come back unnoticed.
A strong closing line: "I never tune the prompt until I've confirmed the right passage reached it."
Last updated 16 October 2026
Sources