What is reranking in search and RAG, and when is it worth it?

A first search has to be fast, so it is a little careless. An EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them. search turns the question into numbers once and compares them with numbers worked out for every passage in advance. It never reads the question and a passage side by side. A reranker does.

Reranking is the second step. Take the 50 or 150 best candidates from the fast search, give each one to a reranking model together with the question, and let it score how well that passage answers. Sort by the new scores and keep the top few for the model that writes the answer.

Why the second step is better

The 2 steps use 2 kinds of model. An embedding model is a bi-encoder: it encodes the question and each passage separately, so the passages can be encoded once, ahead of time. A reranker is usually a cross-encoder: it reads question and passage as one input, so every word of one can be weighed against every word of the other. It can see that "My mixer stopped and shows E04" is answered by the page about error E04, where an embedding search put the page about error E07 first, because both are about mixer errors.

That care costs time. A cross-encoder must run once for every pair, at question time. The Sentence Transformers documentation puts scoring every pair among 10,000 sentences at about 65 hours, against 5 seconds to embed them. So nobody reranks a whole library. The fast search narrows it down, and the reranker orders the shortlist.

What it adds

In Anthropic's tests, a reranking step on top of hybrid searchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found. took failed searches from 49% fewer to 67% fewer than plain embeddings. Anthropic fetched 150 candidates and kept the 20 the reranker liked best.

The cost is a model call on every question, and the LatencyLatencyHow long a user waits for a model's answer, measured as the time to the first token and the time to the last, and judged by the slowest requests, not the average. that comes with it. More candidates give the reranker more chances to find the right passage, and take longer. Measure both: recall at kRetrieval EvaluationTesting the search step of a RAG system on its own, with questions whose answering passages you know, and scoring it with recall at k, the share of those passages found in the top k results. for the gain, and the time each question takes.

Which reranker

Hosted rerankers take a question and a list of passages and return them in order: Voyage AI, the embedding company Anthropic's documentation points to, has rerank-3 and a faster rerank-3-lite, and other companies sell them too. Open ones run on your own machine. cross-encoder/ms-marco-MiniLM-L6-v2 is small, Apache 2.0 licensed and runs on a laptop CPU through the Sentence Transformers library:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
scores = reranker.predict([(question, passage) for passage in candidates])

A language model can rerank too, by being asked which passages answer the question. It is slower and dearer than a cross-encoder, so try a dedicated reranker first.