# What is query rewriting in RAG, and when does it help?

Having a model turn what the user typed into a better search query before retrieval, most often turning a follow-up like "and the bigger one?" into a full question the search can match.

- Page: https://tecyfy.com/concepts/query-rewriting
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Query Reformulation, Query Transformation, Standalone Question, HyDE, Hypothetical Document Embeddings

People do not type search queries. They type messages. A search engine can only work with the words it is given, and in a conversation those words are often not enough.

Query rewriting puts a model between the user and the search. It reads what the user wrote, and the conversation so far, and writes the query the search should run.

## The follow-up problem

The most common case is a follow-up. A customer asks "Does the Aster 26 cm pan work on induction?", gets an answer, and then asks "And the copper one?". Searched on its own, "And the copper one?" finds nothing useful: no product, no hob, no question. A person reading the chat knows it means "Does the Copper Classic pan work on induction?". So can a model.

The rewrite step gives the model the recent messages and asks for one standalone question, with every name spelled out. That question goes to the search. The original message still goes to the model that writes the answer, so the reply sounds like part of the conversation.

## Other rewrites

- **Splitting.** "What does express delivery cost and can I return a used pan?" asks 2 things that live on 2 pages. Search for each part and merge the results.
- **Expanding.** Add the words your documents use: a customer's "money back" is the help pages' "refund". A rewrite prompt that knows your vocabulary can bridge it, though [hybrid search](https://tecyfy.com/concepts/hybrid-search) already catches much of this.
- **Writing a draft answer.** A method called HyDE has the model write a short answer it imagines, then searches with that answer instead of the question. A made-up answer reads more like the real passage than a question does, so their [embeddings](https://tecyfy.com/concepts/embedding) sit closer. The draft may be wrong; it is only used to search, never shown.

Research on rewriting has gone further, training a small model to write the queries that lead the answering model to the best results. Most apps do well with a prompt.

## The cost, and how to tell if it pays

Each rewrite is an extra model call before the search, which adds time to every answer. Use a small, fast model, such as [Claude Haiku 5.5](https://tecyfy.com/prices/claude-haiku-5-5), and skip the step where it cannot help: the first message of a conversation, if it is already a full question, can go straight to the search.

Then measure. Take questions from real conversations, follow-ups included, and score the search with and without the rewrite, using [recall at k](https://tecyfy.com/concepts/retrieval-evaluation). Keep the rewrite if recall goes up on the questions people ask, and look at the ones it made worse, because a rewrite can also drift from what the user meant.

## Related

- [RAG](https://tecyfy.com/concepts/rag): A way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes.
- [Hybrid Search](https://tecyfy.com/concepts/hybrid-search): Running a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found.
- [Retrieval Evaluation](https://tecyfy.com/concepts/retrieval-evaluation): Testing the search step of a RAG system on its own, with questions whose answering passages you know, and scoring it with recall at k, the share of those passages found in the top k results.
- [Embedding](https://tecyfy.com/concepts/embedding): A list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them.
- [Context Window](https://tecyfy.com/concepts/context-window): Everything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out.

## Sources

- [Query Rewriting for Retrieval-Augmented Large Language Models](https://arxiv.org/abs/2305.14283)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://arxiv.org/abs/2212.10496)
- [Models overview](https://platform.claude.com/docs/en/about-claude/models/overview)