How RAG search works: embeddings, hybrid search, reranking

The water just sits there

Watch this part · 1:11How Does AI Find a Page When Not One Word Matches? Semantic Search

2 notes. "The water just sits there." And "drain blocked." They share no word at all. Yet type the first into the search this lesson builds, and the first card it returns from our washer manual says: "Water left in the drum at the end. Drain filter or drain hose blocked. See E21."

How does a computer know those 2 mean the same thing? That is the subject here.

What is RAG? followed one question through the 4 steps of RAGRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes.: prepare the documents once, retrieve the right pieces for each question, add them to the question, generate the answer. This lesson is about the first 2, how the pieces are made and how the right ones are found. 2 of the 3 ways RAG goes wrong happen here.

Same manual, the Sudsley S7, and the same question: "My washer shows E21. What do I do?" We ran every search on 8 October 2026, with Google's gemini-embedding-001 for meaning search, and the results below are the ones we got, the misses included. We cut the manual 2 ways: 60 big cards, one per page, and 237 small ones, one per table row or paragraph. Most results use the small cards, and we say when one uses the big ones.

Search by words

Watch this part · 1:38How Did Search Work Before AI? Keyword Search and BM25 Explained

The oldest way to find a page is the index at the back of a book: look up a word, get the pages it is on. A computer can do the same with every word of every card. That is keyword search, and the list of which words are on which cards is a keyword index.

It is a little smarter than a book index. A card scores higher when it has more of your words, when those words are rare, and when the card is short, so your words make up more of it. The standard formula for this is called BM25.

For some questions it is perfect. We searched for "E23", and of 237 cards exactly one contains it: the E23 row of the error table. A part number works the same way: "SP-3021" goes straight to the drain filter in the parts list.

Now the customer's own words, "the water just sits there". Keyword search drops little words like "the" and "there", and of what is left, the manual contains only "water". So the top card is one that says "water" 3 times: "E10. No water. The tap is closed." The opposite problem.

Keyword search matches words, not meaning. If you do not use the manual's words, it cannot help you.

A map of meaning

Watch this part · 1:39What Is an Embedding? How AI Turns Your Words Into 3,072 Numbers

Picture a map where every phrase has a place, and phrases that mean similar things sit close together. A toy version has 2 directions: left to right, how much it is about water; bottom to top, how much it is about something going wrong. "Drain blocked" sits high on the right. "The water just sits there" lands right next to it. "A quick 30-minute wash" is far away in a corner. They are neighbours because of what they mean, with no shared word needed.

A place on a 2-direction map is 2 numbers. The real map has far more directions: Google's embedding models use 3,072 by default, so every phrase becomes a list of 3,072 numbers. That list is the EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them., each direction is a dimensionEmbedding DimensionNumber of elements in an embedding vector., and the program that turns text into the list is an embedding model.

You cannot read the numbers like words. No single one of them stands for "water" or "blocked". The meaning is in the whole list together.

Other models use other lengths. OpenAI's text-embedding-3-small uses 1,536 by default. What matters is that the question and the cards go through the same model, so they land on the same map.

How close is close?

Watch this part · 1:32Cosine Similarity Explained: How AI Measures If Two Sentences Mean the Same

Draw an arrow from the centre of the map to each phrase. Phrases with similar meanings point in nearly the same direction, so the search compares directions: the smaller the angle between 2 arrows, the closer the meaning. The measure is cosine similarityCosine SimilarityA measure of how closely 2 vectors point the same way, from 1 for the same direction to −1 for opposite; with embeddings, a higher score means closer in meaning.. It looks only at the direction of the arrow, not its length. A score of 1 means the same direction, and the lower the score, the further apart. It works the same with 3,072 directions as with 2; nobody can draw that.

Our 2 notes, "the water just sits there" and "drain blocked", scored 0.829. Against "the door is locked", the customer's note scored 0.788. Lower, so the drain note is the closer one, and the search gets it right.

The numbers themselves need care. "Washing machine" and "banana" still scored 0.806, while "washing machine" and "washer", the same thing, scored 0.957. Almost everything landed between 0.7 and 1, so a score on its own tells you little. What counts is the order: for one question, which card is closer than which.

That is all the search does. It scores every card against the question, puts them in order and takes the top few. One correction to the first lesson: there the 2 embeddings looked almost identical. They are not. They are closer than the others, and closer is all a search needs.

Where the map comes from

Watch this part · 1:04Who Teaches AI What Words Mean? How Embedding Models Learn

Nobody places the phrases by hand. The embedding model learns where they go. Google's paper on its Gemini Embedding model describes the training: each example has a question, a right answer and, in some examples, a wrong answer that looks a lot like the right one. Training works by contrast, so with each example the question and the right answer are drawn closer together and the look-alike is pushed away.

Repeat that over an enormous amount of text, and things that answer each other end up close together. If complaints like "the water just sits there" kept being answered by text like "drain blocked", the 2 end up near each other, and nobody wrote that rule. That last part is our illustration, since no one outside the maker can see the training data. The point holds: an embedding model has learned what tends to answer what, and that is how a customer's words find the manual's words.

Where meaning search is blind

Watch this part · 1:36Why AI Search Can't Tell E21 From E23 (Embedding Blind Spot)

Meaning search is good with everyday words, but it can miss exact matches such as codes and part numbers. To our embedding model, "E21" and "E23" scored 0.934, closer than "E21" and "drain blocked", which is what E21 means, at 0.783.

Big cards make it worse. On the 60 page-sized cards, a meaning search for the part number SP-3021 put the empty Notes page at the back of the manual first, and the parts list second. Keyword search found the parts list at once, because it is the only card with that exact word.

Each search misses what the other finds. Fix 1 is to run both and merge the 2 lists, which is hybrid searchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found.. The merge is simple: each card gets points for how high it ranks on each list, so a card near the top of either list ends up near the top. On our manual, hybrid search put E23 and SP-3021 straight back on top. Some ready-made tools do it for you: OpenAI's file search uses both semantic and keyword search.

Hybrid search has a cost of its own, and it shows up 2 sections down.

Cutting the cards

Fix 2 is about the cards themselves, the chunks. The first lesson skipped a question: how big should a card be?

On the 60 big cards, our question "My washer shows E21. What do I do?" brought the error-table page back third, behind the drain-filter page and the page on installing the drain hose. A whole page is about many things, so its embedding is a blur of all of them, and the E21 row is a small part of it. (The first lesson showed the error table coming back first. In the real search it came third, still in the top 3, so the answer would still have worked.)

On the 237 small cards, the E21 row came back first. Sharp. But the customer's words, "the water just sits there", brought back a card that says only "Tap. Water is filling." It is one line from the table of display symbols, and on its own all it says is "water". The card has lost its context: which page, which table, what it is for.

That is the trade-off. Big cards blur. Small cards forget where they came from.

So fix 2 has 2 parts. First, cutChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file. along the document's own structure, a table row or a section under a heading, so a cut never lands in the middle of something like the E21 row. Builders also let neighbouring cards overlap a little. Size, where the cuts fall and overlap can all change what the search finds.

Second, before each card is embedded, give it a short note saying where it comes from. Anthropic calls this contextual retrieval. Our note was simple: the manual's name, the page and the page's heading. Anthropic's version has a model write a note of usually 50 to 100 tokens for every chunk.

With notes on our 237 small cards, both questions came back right. "My washer shows E21" returned the E21 row first, and "the water just sits there" returned "Water left in the drum at the end. Drain filter or drain hose blocked." first. In Anthropic's tests across several kinds of documents, its model-written notes cut failed searches by 35%, and by 49% with keyword search run on the noted chunks too.

A second opinion

Now put fixes 1 and 2 together: small cards with notes, searched with hybrid search. For our E21 question, the E21 row slipped from first to fifth. Keyword search favours the cards that repeat "drain filter" and "E21", and the merge pulled 4 of them above it.

The reason is that both searches are built for speed. One counts words, the other compares lists of numbers made in advance. Neither reads the question next to the card.

Fix 3 is a second step, a rerankerRerankingA second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order.. It takes the best results from the first search, reads the question together with each card, scores how well that card answers it and puts them in a new order. Only the best few go to the model. Reading takes time, so it adds a small delay, which is why it usually runs on a short list: the more cards it reranks, the slower and dearer it gets.

We gave hybrid search's top 20 to a small, cheap Gemini model with one job: order these cards by how useful each is for the question. The E21 row went from fifth back to first, with the drain-filter card from page 41 second.

2 notes on that. Ours was a general model doing the job, and dedicated reranking models are built for it alone. And a reranker can only reorder what the first search found: if the right card is not in the 20, it cannot bring it back. We also ran the whole test twice, and the top card came out the same in 17 of 18 searches while some lower places swapped. A model used as a reranker is not perfectly repeatable.

In Anthropic's tests, adding a reranker to the notes and keyword search cut failed searches by 67%. It took the top 150 results, reranked them and kept 20.

The search, rebuilt

Prepare once: cut the documents along their structure, add a short note to each chunk, embed each one, and store them in a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning. and a keyword index. Then for every question: run keyword search and meaning search and merge the 2 lists, let a reranker read the question with each of the top results and reorder them, and send the top few, the top-k, to the model with the question.

A quick test. With small cards and no notes, the customer's words found "Tap. Water is filling." Which fix stops a card like that from winning in the first place? The context note: once the card says it comes from the table of display symbols, it drops out of the top 5. A reranker pushed it down too, but only after the search had already picked it.

Bad answers often start with bad search

When a RAG app gives a bad answer, look at what the search handed over before you blame the model. In the first lesson, all the model needed was the right page. Get the right card to the top, and the answer follows.

So when you build or buy one, ask what it searches with: words, meaning or both, how it cuts the documents, and whether anything reads the question next to the results. Build a RAG puts that to the test, a ready-made tool against a pipeline built from the pieces in this lesson, on the same manual and the same question.