Embeddings and vector search in Python: semantic, keyword and hybrid search

Every GenAI app that answers from your documents starts with a search. Before the model writes a word, something has to find the 3 passages, out of thousands, that hold the answer. If the search misses, the best model in the world answers from the wrong page. This module builds that search. Module 5 puts a model on top of it.

What an embedding gives you

An EmbeddingEmbeddingA list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them. model reads a piece of text and returns a list of numbers, a vector, that stands for what it means. Texts with similar meaning get vectors that point in similar directions, and cosine similarityCosine SimilarityA measure of how closely 2 vectors point the same way, from 1 for the same direction to −1 for opposite; with embeddings, a higher score means closer in meaning. measures how similar. "How long until I get my money back?" and "We refund within 5 working days" share no useful words, and their vectors are still close.

Try itTurn and stretch two vectors

Angle between the arrows40°

Length of B

Same as A1.8 times

Cosine similarity

0.77

Dot product

0.77

Distance between tips

0.68

That is all a semantic search is: embed every passage once, embed each question as it arrives, and return the passages whose vectors are closest. The trick that makes it fast is the first half. A model that read the question and each passage together would be more accurate, but the Sentence-BERT paper measured the cost: about 65 hours to find the closest pair among 10,000 sentences that way, against about 5 seconds with embeddings worked out once and compared. A sentence embeddingSentence EmbeddingOne list of numbers for the meaning of a whole sentence or passage, made so that texts which mean the same thing get lists close together. It is what semantic search compares. model is the second kind.

Choose an embedding model

You have 2 kinds to choose from. Hosted models are an API call: Anthropic makes none and its documentation points to Voyage AI, and OpenAI and Google sell them too. Open models, such as BAAI/bge-small-en-v1.5, run on your own machine for free through the Sentence Transformers library. This module uses the open one, so every line runs without another account.

What to weigh:

• Quality on your text. Leaderboards test someone else's documents. Embed yours, ask 20 real questions, and count how often the right passage comes first.
• Vector length. bge-small returns 384 numbers, voyage-4 returns 1,024 by default. Longer vectors can hold more, and cost memory: at 4 bytes a number, a million 1,024-number vectors take about 4 GB.
• Longest input. bge-small reads 512 tokens and cuts the rest; Voyage's current models read 32,000. A chunk longer than the limit is searched by its first part only.
• Price and where it runs. The calculator compares every hosted model.

ToolRAG and embedding costWhat a RAG system costs a month

2 details catch people. Many models want questions marked differently from passages: bge puts the line Represent this sentence for searching relevant passages: before the question only, and Voyage takes input_type="query" or "document". And the choice locks you in: vectors from 2 models live in different spaces and cannot be compared, so changing models means embedding everything again. Keep the chunk text, not only the vectors, so that rerun is a script.

Cut documents into chunks

You do not embed whole documents. One vector for a 6-page help centre has to stand for delivery, refunds and warranty at once, and matches every question a little. Cut each page into chunksChunkingCutting documents into smaller passages before they are embedded for retrieval, so a search returns the part that answers a question rather than a whole file. that each answer one kind of question, and the search can return the one that matters.

Try itCut a policy into chunks

Chunk size

25 words50100

Overlap

NoneA quarterHalf

The policy, 148 words

123456

Open a chunk

123456

Chunk 3 · words 51 to 75 · part of the answer

business class needs a director to approve it before booking. Hotels should be close to where you are working. A hotel night in London may

Cut along the document's own structure when it has one. Help pages have headings, so one chunk per section keeps each answer whole. Put the page title and the section heading at the top of every chunk: a section that only says "Order by 2pm on a working day..." does not say it is about express delivery, and its vector will not either. Module 5 goes further and has a model write that context.

Search by meaning

With the vectors normalised to length 1, cosine similarity is a dot product, and comparing a question with every chunk is one matrix product:

question = EMBEDDER.encode(QUERY_PREFIX + "Can you send my order to France?", normalize_embeddings=True)
scores = vectors @ question                 # one score per chunk, vectors being every chunk embedded
best = scores.argsort()[::-1][:3]           # the 3 highest

This is exact search: it compares with every chunk. On the laptop this course was tested on, a million vectors of 384 numbers took about 25 milliseconds. Past that, or when every millisecond counts, an approximate index such as HNSWHNSWHierarchical Navigable Small World: a vector index that links each vector to its near neighbours in layers of graphs, so a search hops from coarse to fine and checks only a small share of the vectors. finds nearly the same top results by visiting a small part of the vectors.

Semantic search is at its best when the customer's words differ from the page's. It is at its worst on exact strings. Ask the shop's help pages "My mixer stopped and shows E04" and bge-small ranks the page on error E07 above the page on E04, because both are pages about mixer errors and the code barely moves the vector.

Search by keyword

Keyword search catches what embeddings blur. The standard method, BM25, scores a chunk for each word of the question it contains, with 3 adjustments: rare words count more than common ones, a word repeated in a chunk adds less each time, and long chunks are evened out against short ones. Most databases and search engines have it built in. It is short enough to write yourself, which is the best way to see what it does:

idf = math.log((n - df + 0.5) / (df + 0.5) + 1)  # df: chunks containing the word, n: all chunks
score += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / average))  # tf: times in this chunk

On "shows E04" it puts the E04 page first, because only one chunk contains e04. Its weakness is the other side. Asked "How long until I get my money back?", it ranks the warranty page first, which matches "how long", and a page about the mixer's bowl second, which contains "until". The refund page never says "money".

Use both: hybrid search

Hybrid searchHybrid SearchRunning a keyword search (usually BM25) and a vector search for the same query and merging the 2 ranked lists, so exact terms like a product code and loose wording like "money back" are both found. runs both searches and merges the lists. Their scores are on different scales, so the standard merge ignores scores and uses positions. Reciprocal rank fusion gives each chunk 1 / (60 + its rank) in each list and adds them up. The 60 comes from the 2009 paper that proposed it, which found the exact value mattered little.

fused = Counter()
for ranking in (by_meaning(question, 20), by_keyword(question, 20)):
    for rank, chunk in enumerate(ranking, start=1):
        fused[chunk["id"]] += 1 / (60 + rank)

Fusion is good at one thing: making sure the right chunk is somewhere in the top 20. It is not good at putting it first. On "shows E04", meaning ranks E07 first and keyword ranks E04 first, so they tie. On "money back", the keyword list's mistakes pull the warranty page to the top. Ordering the shortlist is a separate job.

Rerank the shortlist

A rerankerRerankingA second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order. reads the question and each candidate together and scores how well that chunk answers it. That is the accurate, slow kind of model the embeddings avoided, and it is affordable now because it reads 20 chunks, not all of them. Anthropic's tests reranked 150 candidates down to 20 and cut failed searches further than hybrid search alone.

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
scores = reranker.predict([(question, f"{c['title']}\n{c['text']}") for c in candidates])

With a reranker after the fusion, the shop's search puts E04 first for "shows E04" and the refund page first for "money back". It is not free: a model runs on every question, which adds time. Module 5 measures whether each step earns its place.

Where the vectors live

For a few thousand chunks, a NumPy array in memory is the whole vector store: 3,000 vectors of 384 numbers take under 5 MB. Move to a vector databaseVector DatabaseA database built to store embeddings and quickly find the ones closest to a query, which is how AI apps search by meaning. when you need what an array cannot give: filters ("only pages for this country"), updating chunks while the app runs, an index shared between servers, or millions of vectors behind an approximate index. If you already run Postgres, the pgvector extension puts the vectors next to the rest of your data before you add a new service.

Comparisonpgvector vs Qdrant vs ChromaStart with pgvector if you already run Postgres - the vectors sit in the same tables, transactions and backups as the rest of your data, and a filter is an… ToolVector database size calculatorDocuments, articles, tickets or products in; the vectors they make and the memory and disk the index needs out.

Build it: search the shop's help pages

This is the search that module 5's assistant answers from. It reads a folder of help pages, cuts them into one chunk per section, and searches 4 ways, so you can see what each one finds.

Install the library. It brings PyTorch with it, and downloads the 2 small models the first time it runs:

pip install sentence-transformers

Make a folder called docs beside triage.py and save these 6 pages in it. They are short on purpose; the same code works on hundreds.

# Delivery

## Delivery times and costs
Standard delivery takes 3 to 5 working days and costs £4.95. It is free on orders over £50.

## Express delivery
Order by 2pm on a working day and express delivery arrives the next working day. It costs £9.95 and is not available for large appliances.

## Where we deliver
We deliver to addresses in the UK and Ireland. We cannot send orders to other countries yet.

## Tracking an order
We email a tracking link when your order leaves our warehouse. If tracking has not changed for 3 working days, contact us with your order number and we will chase the courier.

That is docs/delivery.md. Then docs/returns.md:

# Returns and refunds

## Returning an item
You can return any unused item within 30 days of delivery. Start a return from your account page and print the free returns label.

## Damaged or faulty on arrival
If an item arrives damaged, tell us within 48 hours and send a photo of the damage. A member of our team checks it and arranges a replacement or a refund.

## When your refund arrives
We refund to your original payment method within 5 working days of receiving the return. Your bank can take a few more days to show it.

docs/warranty.md:

# Warranty

## What the warranty covers
Every product has a warranty against faults in materials and manufacture. It does not cover wear and tear, accidental damage or misuse.

## How long each product is covered
Electrical appliances are covered for 2 years. Stand mixers are covered for 5 years on the motor and 2 years on every other part. Knives have a lifetime warranty.

## Making a warranty claim
Send us your order number, a description of the fault and a photo or short video. If we confirm a fault, we repair or replace the product.

docs/cookware.md:

# Pans and cookware

## Aster 26 cm frying pan (AST-26)
An aluminium frying pan with a non-stick coating. It works on every hob, including induction, and is oven safe up to 220°C once the silicone handle sleeve is removed. Wash it by hand.

## Aster 28 cm frying pan (AST-28)
The larger Aster pan, with the same non-stick coating and a helper handle. It works on every hob, including induction, and is oven safe up to 220°C. It can go in the dishwasher on a gentle cycle.

## Copper Classic 28 cm pan (CC-28)
A copper pan lined with stainless steel. It does not work on induction hobs. It is oven safe up to 260°C. Wash it by hand and polish the copper now and then.

## Caring for non-stick pans
Use wooden or silicone utensils and medium heat. Metal utensils and very high heat wear the non-stick coating faster.

docs/knives.md:

# Knives

## Cleaning your knives
Wash knives by hand in warm soapy water and dry them straight away. Dishwashers can blunt the blade and leave spots on the steel.

## Sharpening
Hone the blade every few uses and sharpen it 2 or 3 times a year. We sharpen knives bought from us for free once a year.

## Lifetime warranty on knives
The lifetime warranty covers faults in the steel and the handle. It does not cover chips from cutting bones or damage from a dishwasher.

And docs/mixer.md:

# Stand mixer help

## Error E01
The bowl is not locked in place. Turn the bowl clockwise until it clicks, then press start again.

## Error E04
The motor is too hot. Switch the mixer off at the wall and leave it for 30 minutes before you use it again. Heavy bread dough at high speed is the usual cause.

## Error E07
The speed sensor has a fault. Stop using the mixer and contact us with your order number, because this fault is covered by the warranty.

## Attaching the tools
The whisk, beater and dough hook push onto the drive shaft and turn a quarter turn to lock. Lift the head before you change a tool.

Now the search. Save it as search.py:

"""Search the shop's help pages by meaning, by keyword, and both, then rerank."""
import math
import re
import sys
from collections import Counter
from pathlib import Path

from sentence_transformers import CrossEncoder, SentenceTransformer

EMBEDDER = SentenceTransformer("BAAI/bge-small-en-v1.5")
RERANKER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
# bge models search better when the question, and only the question, starts with this line.
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "


def chunk(path: Path) -> list[dict]:
    """One chunk per ## section, its page title and heading in front, so it says what it is about."""
    page, chunks = "", []
    for block in re.split(r"^## ", path.read_text(), flags=re.MULTILINE):
        if block.startswith("# "):
            page = block[2:].splitlines()[0].strip()
            continue
        heading, _, body = block.partition("\n")
        slug = re.sub(r"[^a-z0-9]+", "-", heading.lower()).strip("-")
        chunks.append({"id": f"{path.stem}#{slug}", "title": f"{page}: {heading.strip()}",
                       "text": body.strip(), "source": f"{path.stem}.md"})
    return chunks


def words(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())


class Index:
    def __init__(self, folder: str = "docs"):
        self.chunks = [c for path in sorted(Path(folder).glob("*.md")) for c in chunk(path)]
        texts = [f"{c['title']}\n{c['text']}" for c in self.chunks]
        self.vectors = EMBEDDER.encode(texts, normalize_embeddings=True)
        self.terms = [Counter(words(text)) for text in texts]
        self.lengths = [sum(terms.values()) for terms in self.terms]
        self.average = sum(self.lengths) / len(self.lengths)
        self.containing = Counter(word for terms in self.terms for word in terms)

    def by_meaning(self, query: str, k: int) -> list[dict]:
        q = EMBEDDER.encode(QUERY_PREFIX + query, normalize_embeddings=True)
        scores = self.vectors @ q  # the vectors have length 1, so this is cosine similarity
        return [self.chunks[i] for i in scores.argsort()[::-1][:k]]

    def by_keyword(self, query: str, k: int, k1: float = 1.2, b: float = 0.75) -> list[dict]:
        """BM25: rare words count for more, repeats count for less, long chunks are evened out."""
        n, scores = len(self.chunks), []
        for terms, length in zip(self.terms, self.lengths):
            score = 0.0
            for word in set(words(query)):
                if word in terms:
                    df = self.containing[word]
                    idf = math.log((n - df + 0.5) / (df + 0.5) + 1)
                    tf = terms[word]
                    score += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / self.average))
            scores.append(score)
        ranked = sorted(range(n), key=lambda i: scores[i], reverse=True)
        return [self.chunks[i] for i in ranked[:k] if scores[i] > 0]

    def hybrid(self, query: str, k: int, pool: int = 20) -> list[dict]:
        """Reciprocal rank fusion: a chunk scores 1 / (60 + its rank) in each list it appears in."""
        fused, by_id = Counter(), {}
        for ranking in (self.by_meaning(query, pool), self.by_keyword(query, pool)):
            for rank, c in enumerate(ranking, start=1):
                fused[c["id"]] += 1 / (60 + rank)
                by_id[c["id"]] = c
        return [by_id[i] for i, _ in fused.most_common(k)]

    def search(self, query: str, k: int = 3, pool: int = 20) -> list[dict]:
        """Hybrid search for a pool of candidates, then a cross-encoder reads each one with the query."""
        candidates = self.hybrid(query, pool)
        scores = RERANKER.predict([(query, f"{c['title']}\n{c['text']}") for c in candidates])
        ranked = sorted(zip(scores, range(len(candidates))), reverse=True)
        return [candidates[i] for _, i in ranked[:k]]


if __name__ == "__main__":
    index = Index()
    query = " ".join(sys.argv[1:])
    for name in ("by_meaning", "by_keyword", "hybrid", "search"):
        found = getattr(index, name)(query, 3)
        print(f"{name:>10}: {', '.join(c['id'] for c in found) or 'nothing'}")

Index embeds every chunk when it starts, which takes a few seconds for these pages. For a real help centre you would embed once, save the vectors with the chunk ids, and embed again only the pages that change.

Try the 2 questions from this module:

python search.py "My mixer stopped and shows E04"
python search.py "How long until I get my money back?"

When we ran it, the first printed:

by_meaning: mixer#error-e07, mixer#error-e04, mixer#error-e01
by_keyword: mixer#error-e04, mixer#error-e07, mixer#error-e01
    hybrid: mixer#error-e07, mixer#error-e04, mixer#error-e01
    search: mixer#error-e04, mixer#error-e07, mixer#error-e01

Meaning search puts E07 first, keyword search gets E04, the fusion ties them, and the reranker gets it right. The second question turns it around: keyword search starts with the warranty page and meaning search finds the refund page. Try your own questions, especially ones in the customer's words rather than the page's, and watch where each method goes wrong. index.search() is the one module 5 uses, and module 5 replaces "try a few questions" with a test that scores every method on the same list.