What is RAG? How AI answers from your own documents

A washer error that no AI could answer

Watch this part · 1:29Why Can't ChatGPT, Claude or Gemini Answer From Your Documents? The Fix

A washing machine stops halfway through a wash and its screen blinks E21. The answer is in the manual, somewhere in 60 pages, in a drawer. So you ask an AI instead: "My washer shows E21. What do I do?"

We tried this with ChatGPT, Claude and Gemini on 7 October 2026. None of them could give the steps for this machine. Claude said it had no reliable information about it. ChatGPT looked online, found nothing and would not say for sure. Gemini was certain: E21 is a drain problem, here are 3 steps. That was a fair guess, but it was built from a general video about washing machines, not from this machine's manual.

They could not have known. The washer, the Sudsley S7, is one we made up, with a manual we wrote ourselves. No model has read it. Your own documents are in the same position: a company handbook, last week's meeting notes, a manual from a small brand. They are private, or newer than the model, or both. A model learned from what was public before its knowledge cutoffKnowledge CutoffThe date a model's training text ends. The model knows nothing after it unless it is given new text, for example by a web search., as How AI learns explains, and your files were never in that pile.

So hand it the right page at the moment you ask. That is RAGRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes..

RAG is an open-book exam

Watch this part · 1:15RAG Explained With One Exam: Open-Book vs Closed-Book AI

2 students sit the same exam. One has the book closed and works from memory, so a question they never learned gets a guess. That is a model on its own. The other has the book open. They read the question, find the right page, read it and write the answer in their own words.

RAG hands the AI the open book. The letters stand for retrieval-augmented generation. Retrieval finds the right pages. Augmented means they are added to the question. Generation is the model writing the answer.

Notice what does not change: the student. Nobody retrains the model, the part that writes the answers. It knows exactly what it knew before and gets the right page at the right moment. That is why RAG is easy to keep current. A new manual next month means new pages, not a new model.

One word from here on. Everything the model reads before it answers, your question and whatever is added to it, is its context. How much fits at once is the context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out..

Why not paste the whole manual?

Watch this part · 1:27Why Not Just Paste the Whole Document Into the AI? (RAG vs Long Context)

For a 60-page manual, pasting it all into the chat often works. Some models now take 1 million tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. or more, and a token is a small piece of text, a few letters long. A useful line for builders: under about 200,000 tokens, roughly 500 pages, you can put the whole thing in the prompt. With cachingPrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker., where the provider keeps what it has already read for a while, sending it again with every question costs much less.

A company's documents are rarely 60 pages. Every manual for every machine it has sold, the help articles, the repair notes: thousands of pages, changing every week. AI is paid for by the token, so pasting all of it into every question means paying for all of it to be read every time. More is not always better either. When a model has to find several specific facts in a huge pile, it gets less accurate.

The rule of thumb: small and stable, hand it everything. Big or changing, hand it only the pages that matter. Finding those pages is RAG's job, in 4 steps.

Step 1: prepare the documents

Watch this part · 1:28RAG Step 1: What Happens to Your Documents Before You Ask Anything

The first step happens once, before anyone asks anything. The manual is cut into pieces of a few paragraphs, like index cards. Builders call them chunks, and the step is ChunkingChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file.. The point is to hand the model later just the part that matters, not the chapter around it. One card holds the error-code table from page 34. Another holds the drain-filter steps from page 41.

Each card then gets an EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them., a code that captures what the card is about. It lets the system find cards by meaning, not only by matching the exact words, which is called semantic search. An embedding is a long list of numbers, a vector, so the cards and their embeddings are filed in a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning., an index built for this job. Cards about similar things get similar embeddings. How that works is the subject of how RAG search works.

A new manual goes through the same step: chunk it, embed it, file it. Nothing is retrained.

Step 2: retrieve the closest pages

Watch this part · 1:16How Does RAG Find the Right Page in a Second? Similarity Search Explained

The second step happens every time someone asks. The question "My washer shows E21. What do I do?" gets an embedding made the same way as the cards'. The system then finds the cards whose embeddings are closest to it, which is similarity searchVector SearchFinding the stored items whose embeddings are closest to a query's embedding, so a search matches meaning rather than exact words.: the 2 lists of numbers are compared, and the more alike they are, the closer the meaning.

It keeps the closest few. How many is a setting builders call top-k, here 3: the error table from page 34, the drain filter from page 41, and page 39 on pump noises, which is close but not what we need. That is normal. The search returns what is closest, not what is correct, and the model sorts out the rest. Because it compares lists of numbers instead of reading pages, it is fast, even across thousands of pages.

Step 3: the message the AI really receives

Watch this part · 1:14What Does an App Really Send to the AI in RAG? The Actual Prompt

This is the step almost nobody shows. The app builds one message, the prompt, in 3 parts. First, instructions: "Answer the customer's question using only the pages below. Say which page you used. If the answer isn't in these pages, say you don't know." Then the 3 cards, each with its page number. Then, at the end, the question.

That is the "augmented" in RAG: the question, the pages and the rules for using them. It is all plain text. The model gets no special access to your files, only what is written into this prompt.

Each rule narrows a gap. "Only from these pages" tells the model not to fill gaps from what it learned in training, which cuts down how often it does. "Say which page" lets you check its work. "Say you don't know" gives it a way out instead of a guess. Leave them out and you have handed over the right pages without asking the model to stick to them.

Step 4: an answer you can check

Watch this part · 1:21How to Check an AI Answer: Grounded Answers With Page Numbers (RAG)

The model writes: "E21 means the water isn't draining. Switch the machine off, open the small flap at the bottom right, and unscrew the drain filter to clear it. See pages 34 and 41." It used 2 of the 3 cards and left the pump page out. Picking what matters from a few pages is something these models do well.

Now you can check it. Page 34 says E21 is "drain blocked". Page 41 has the filter steps. An answer built from pages you supplied, and traceable back to them, is groundedGroundingMaking a model answer from sources you give it, such as retrieved documents, and only from them, so every statement can be traced to a passage and a missing answer is admitted, not invented.. At the start the same kind of model gave a polite "I don't know" or a confident guess from other machines. The difference is these 4 steps.

When we gave the same 3 apps the whole manual, all 3 got E21 right. For 60 pages, handing over everything works. For thousands, you need the 4 steps.

Where you already use RAG

Watch this part · 1:00Where You Already Use RAG: Claude Projects, ChatGPT Files, Help Chats

Claude projects make the paste-or-search choice for you. While a project's files fit in the context window, Claude reads all of them. As they approach the limit, it switches on its own to a search tool that retrieves just the relevant parts, and a project can then take in up to 10 times as much. ChatGPT also takes uploaded files and answers questions about them.

If you build apps, you do not have to make every step yourself. Gemini's File Search and OpenAI's file search do the cutting, embedding and searching for you, and their answers come with citations, like our page numbers. Build a RAG compares that with building your own.

A help chat that answers and points you to an article may well be doing the same thing.

3 ways RAG still gets it wrong

Watch this part · 1:203 Ways RAG Still Gets It Wrong (and How to Spot Them)

RAG is a large improvement and not magic. It fails in 3 ways, and knowing them is most of the skill.

The wrong cards come back. If the search returns the pump page, the dryer page and the warranty page, the model never sees E21. At best it says it does not know. At worst it guesses.

The right page was cut badly. If the cut falls through the E21 row, one card says "E21" and the next says "drain blocked". Neither makes sense alone, and the search may find only one.

The answer goes beyond the pages. The model adds a step from memory that is not in your manual, so the answer is no longer grounded. A statement that is wrong, or does not match the pages the model was given, is called a HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.. The rules and the page numbers are there so you can catch it.

The first 2 are search problems, which is why the search gets a lesson of its own.

The 4 steps, and a quick test

Watch this part · 59 sRAG in 60 Seconds: The 4 Steps Explained + a Quick Quiz

Prepare once: chunk the documents, embed each chunk, store them in a vector database. Retrieve: embed the question and pull back the closest chunks. Augment: instructions, chunks and question go to the model as one prompt. Generate: the model writes a grounded answer with page numbers you can check.

A new question arrives: "How do I set a quick 30-minute wash?" Which step finds the right page? Step 2, retrieve. And which step makes the model say "I don't know" when that page is missing? Step 3, through the instructions it was given.

RAG makes AI informed, not smarter

Watch this part · 48 sDoes RAG Make AI Smarter? No, It Makes It Informed

RAG does not make a model smarter. It makes it informed. Most of the time the answer you need is not something the model should know. It is something your documents already say.

So when an app answers from your files, read it the way you would read a colleague's summary: which pages did it use, and do they say that? An answer with no page you can open is a guess until you check it. An answer that says the pages do not cover your question is the system working as it should.