How to build a RAG app that answers from your own documents, step by step in Python
A RAG app does 2 jobs: once, it cuts your documents into chunks and stores their embeddings; for each question, it finds the closest chunks and asks a model to answer from them only, with sources. Here is a working version in one Python file, grown one step at a time, and the 4 places these apps usually fail.
Tecyfy · Published · 10 min read

To build a RAG app, you write 2 small programs. The first runs once: it cuts your documents into chunks, turns each chunk into an embedding and stores them. The second runs for every question: it embeds the question, finds the closest chunks, and asks a model to answer only from those chunks, naming its sources. The whole thing fits in one Python file of about 70 lines, with no vector database to start.
Retrieval-augmented generationRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes. is the standard way to make a model answer from your contracts, wiki or tickets instead of from memory. Below, the file grows one step at a time, then the last section covers where these apps usually go wrong.
The pipeline, one question at a time
Step through what happens to the question "How long do refunds take?" on its way to an answer:
A visitor asks a question in plain words. Nothing has been searched yet.
The chunks and scores are an example. Real scores depend on the embedding model; what matters is their order.
Step 1: load and chunk
Cut each document along its own structure, headings first, then paragraphs, so a chunk is about one thing. Keep the title and heading with every chunk, so a chunk about "the limit" still says which limit.
from pathlib import Path
def chunks_from(path: Path, max_words: int = 250):
title = path.stem.replace("-", " ")
section, words, out = "", [], []
for line in path.read_text().splitlines():
if line.startswith("#"):
if words:
out.append((section, " ".join(words)))
section, words = line.lstrip("# "), []
continue
words += line.split()
if len(words) >= max_words:
out.append((section, " ".join(words)))
words = []
if words:
out.append((section, " ".join(words)))
return [
{"source": f"{path.name} > {sec}", "text": f"{title}. {sec}. {body}"}
for sec, body in out
]
docs = [c for p in Path("docs").glob("*.md") for c in chunks_from(p)]
250 words is roughly 330 tokens, a common size, since RAG chunks are usually no more than a few hundred tokens; what chunk size for RAG has the evidence and a way to test yours.
Step 2: embed and store
Each chunk becomes a vector. OpenAI's text-embedding-3-small is cheap and returns vectors already scaled to length 1, so the dot product is the cosine similarity.
import numpy as np
from openai import OpenAI
oai = OpenAI()
def embed(texts):
out = []
for i in range(0, len(texts), 100):
r = oai.embeddings.create(model="text-embedding-3-small", input=texts[i:i + 100])
out += [d.embedding for d in r.data]
return np.array(out, dtype=np.float32)
vectors = embed([c["text"] for c in docs])
np.save("vectors.npy", vectors)
A few thousand chunks fit in memory with room to spare. Save the array and the chunk list, and indexing is done until the documents change. Which model to use is its own decision: how to choose an embedding model.
Step 3: retrieve
Embed the question the same way and take the closest chunks.
def search(question: str, k: int = 5):
q = embed([question])[0]
scores = vectors @ q
best = np.argsort(-scores)[:k]
return [docs[i] | {"score": float(scores[i])} for i in best]
Step 4: answer from the chunks, with sources
The prompt does 3 things: gives the passages with their sources, tells the model to use only them, and tells it what to say when they do not hold the answer.
import anthropic
claude = anthropic.Anthropic()
SYSTEM = (
"Answer only from the passages provided. After each claim, cite its source "
"in square brackets. If the passages do not contain the answer, say you "
"could not find it in the documents. Do not use outside knowledge."
)
def answer(question: str):
hits = search(question)
passages = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
reply = claude.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
system=SYSTEM,
messages=[{
"role": "user",
"content": f"<passages>\n{passages}\n</passages>\n\nQuestion: {question}",
}],
)
return "".join(b.text for b in reply.content if b.type == "text")
print(answer("How long do refunds take?"))
That is a working RAG app. The response is read by block type, because newer Claude models can start with a thinking block. For citations checked by the API rather than by the prompt, Claude's citations feature, with each chunk sent as a document with citations enabled, returns the exact passages that support each claim.
Where RAG apps fail, and the fix for each
1. The right chunk never comes back. This happens often, and no prompt can fix a passage that was never retrieved. Write 30 to 50 real questions with the passage that answers each, and count how often that passage is in the top 5. If it often is not, fix retrieval before anything else.
2. Exact words get lost. Embeddings match meaning, so they can miss a product code, an error number or a name. Run a keyword search beside the vector search and merge the results (hybrid searchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found.). Plain keyword search beside embeddings already helps. In Anthropic's tests, having a model add 50 to 100 tokens of context to each chunk before embedding cut failed top-20 retrievals by 35%, and by 49% when the keyword index was built from those contextualised chunks too.
3. Good chunks rank too low. Fetch a large set of candidates, then let a rerankerRerankingA second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order. keep the best few. In Anthropic's tests, reranking the top 150 down to 20, on top of contextual embeddings and contextual keyword search, cut failed top-20 retrievals by 67%.
4. The model answers from memory. If it answers when the passages are silent, make the instruction to say "not found" explicit, as above, and test with questions your documents cannot answer.
When you need more than a NumPy array
Move to a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning. when the vectors outgrow memory, when several servers must share them, or when you need filters (only this customer's documents) and updates without a rebuild. pgvector vs Qdrant vs Chroma compares the common choices, and the vector database size calculator shows the memory each needs. And if all your documents together are under about 200,000 tokens, Anthropic notes you can skip retrieval and put them all in the prompt, with prompt caching keeping the cost down.
Questions
- How do you build a RAG application?
- In 2 parts. Indexing, done once: load your documents, cut them into chunks of a few hundred tokens along headings and paragraphs, turn each chunk into an embedding and store it with its text and source. Answering, for each question: embed the question, find the closest chunks, and send them to a language model with an instruction to answer only from them and cite the source.
- Do I need a vector database to build RAG?
- Not to start. A few thousand chunks fit in memory, and a NumPy array with a dot product finds the closest ones in milliseconds. Move to a vector database such as pgvector or Qdrant when the collection grows past what fits in memory, when several processes must share it, or when you need filters and updates.
- Why does my RAG app give wrong answers?
- Often because the right passage was never retrieved. Check retrieval first: for a set of real questions, is the passage that answers each one in the top 5? If not, fix chunking, add keyword search alongside embeddings, or add a reranker. If it is, tighten the prompt so the model answers only from the passages and says when they do not contain the answer.
- How many chunks should RAG retrieve?
- Measure on your own questions. In Anthropic's tests, passing 20 chunks worked better than 5 or 10, though more text can also distract the model, so there is a limit. Too few and an answer spread across sections is missed; too many and the prompt fills with text the model must read past and you pay for. With a reranker, fetch a large set by embedding and keep the best few. Anthropic's tests fetched 150 and kept 20.
Sources