Why use hybrid search and reranking in a RAG system?

Vector search finds passages with similar meaning but can miss exact terms such as an error code, a product id or a name. Keyword search (BM25) finds exact terms but misses paraphrases. Hybrid search runs both and merges the results, so each covers the other's blind spot. A reranker then reads the question with each candidate and reorders them by true relevance, so the best few go in the prompt. Together they are the cheapest large gain in most RAG systems.

Why vectors alone fall short

Ask "What does error TS-999 mean?" An embedding captures "an error code in a technical context", and nearby vectors may be about other error codes. BM25 matches TS-999 exactly. Anthropic's contextual retrieval write-up uses this very example.

Merging the two lists

The scores are on different scales, so simple addition does not work. Reciprocal rank fusion (RRF) uses only rank positions: each document scores the sum over lists of 1 / (k + rank), with k = 60 by default in Elasticsearch. A document near the top of both lists wins, with no tuning needed. Weighted score blending also works, but needs calibrating.

Reranking

• Retrieve broadly, for example the top 50 to 150 from hybrid search.
• A reranker (a cross-encoder model, or a hosted rerank API) scores each pair of question and passage together. That is much more accurate than comparing two separate vectors, but too slow to run over the whole index.
• Keep the top 10 to 20 for the prompt.

What it is worth (Anthropic's measurements)

Starting from a top-20 retrieval failure rate of 5.7%:

• contextual embeddings: 3.7% (35% fewer failures)
• plus contextual BM25: 2.9% (49% fewer)
• plus reranking: 1.9% (67% fewer)

They also found that passing the top 20 chunks did better than the top 10 or 5.

Costs to mention

• Reranking adds latency (a model call per query) and cost. Measure whether the recall gain is worth it.
• Two indexes to keep in sync with the documents.
• Filters (permissions, dates, tenants) must apply to both searches before merging, or one path leaks documents the user may not see.

How to show you would build it

"I'd measure recall@20 on a labelled question set for vector only, BM25 only, hybrid with RRF, and hybrid plus rerank, then keep the cheapest setup that meets the target."