In a RAG chatbot, how do you handle follow-up questions such as "what about the second one?"

A follow-up like "what about the second one?" means nothing on its own, so searching with it retrieves the wrong passages. Rewrite the question into a standalone search query using the conversation ("What is the return policy for the Pro plan?"), retrieve with that, and answer with both the recent conversation and the new passages in the prompt. Then evaluate on real multi-turn conversations, not only single questions.

The problem

Retrieval matches the query against your documents. "Does it work offline?" has no product name; "and in Germany?" has no topic. The model generating the answer can see the conversation, but the search index cannot.

The fix: query rewriting

1. Take the last few turns and the new message.
2. Ask a small, fast model to write a standalone question: "Given this conversation, rewrite the user's last message as a self-contained search query. If it is already self-contained, return it unchanged."
3. Retrieve with the rewritten query (hybrid search and reranking as usual).
4. Generate the answer with the original message, the recent conversation and the retrieved passages.

Log the rewritten query with every answer: when an answer is wrong, it is often the first thing to check.

Variations

• Several queries for broad questions ("compare plan A and plan B" → one query per plan), then merge the results.
• Skip retrieval when the answer is already in the conversation ("can you shorten that?"). A small classifier or the rewriting step can decide.
• Keep earlier passages for a few turns, so a follow-up about the same document does not need a new search.

Managing the conversation

• Keep the last few turns verbatim and summarise older ones, so the prompt does not grow without limit.
• Drop old retrieved passages that are no longer relevant.
• Keep the user's permissions applied to every retrieval, not only the first.

Evaluating it

Single-question test sets miss this entirely. Add conversations of 3 to 5 turns with follow-ups, pronouns and topic changes, and check:

• the rewritten query (does it contain the missing subject?),
• retrieval recall for the rewritten query,
• the final answer.

Cost and latency

Rewriting adds a model call. Use a small model, run it in parallel with other preparation where possible, and skip it when the message is clearly self-contained.

What interviewers look for: knowing that retrieval needs a standalone query, and that multi-turn behaviour has to be tested on its own.