# Embeddings and vector search in Python: semantic, keyword and hybrid search

Embeddings turn each passage of your documents into a list of numbers that stands for its meaning, so a search can find the passages closest to a question even when they share no words. On its own that misses exact terms like a product code or an error number, so a GenAI app usually runs keyword search (BM25) beside it, merges the 2 ranked lists with reciprocal rank fusion, and lets a reranking model put the shortlist in order before the passages go to the model.

- Page: https://tecyfy.com/ai-engineering/embeddings-and-search
- Course: AI engineering course (https://tecyfy.com/ai-engineering), Module 4: Embeddings and search

At the end of this module you can:

- Turn text into embeddings with an open or a hosted model, and know what locks you in
- Cut documents into chunks a search can return whole
- Search by meaning, and by keyword with BM25, and see what each one misses
- Merge both with reciprocal rank fusion and rerank the shortlist with a cross-encoder
- Choose where the vectors live, from an array in memory to a vector database

Every GenAI app that answers from your documents starts with a search. Before the model writes a word, something has to find the 3 passages, out of thousands, that hold the answer. If the search misses, the best model in the world answers from the wrong page. This module builds that search. Module 5 puts a model on top of it.

## What an embedding gives you

An [Embedding](https://tecyfy.com/concepts/embedding) model reads a piece of text and returns a list of numbers, a vector, that stands for what it means. Texts with similar meaning get vectors that point in similar directions, and [cosine similarity](https://tecyfy.com/concepts/cosine-similarity) measures how similar. "How long until I get my money back?" and "We refund within 5 working days" share no useful words, and their vectors are still close.

(An interactive demo goes here on the page: https://tecyfy.com/ai-engineering/embeddings-and-search)

That is all a semantic search is: embed every passage once, embed each question as it arrives, and return the passages whose vectors are closest. The trick that makes it fast is the first half. A model that read the question and each passage together would be more accurate, but the Sentence-BERT paper measured the cost: about 65 hours to find the closest pair among 10,000 sentences that way, against about 5 seconds with embeddings worked out once and compared. A [sentence embedding](https://tecyfy.com/concepts/sentence-embedding) model is the second kind.

## Choose an embedding model

You have 2 kinds to choose from. Hosted models are an API call: Anthropic makes none and its documentation points to Voyage AI, and OpenAI and Google sell them too. Open models, such as `BAAI/bge-small-en-v1.5`, run on your own machine for free through the Sentence Transformers library. This module uses the open one, so every line runs without another account.

What to weigh:

- **Quality on your text.** Leaderboards test someone else's documents. Embed yours, ask 20 real questions, and count how often the right passage comes first.
- **Vector length.** bge-small returns 384 numbers, voyage-4 returns 1,024 by default. Longer vectors can hold more, and cost memory: at 4 bytes a number, a million 1,024-number vectors take about 4 GB.
- **Longest input.** bge-small reads 512 tokens and cuts the rest; Voyage's current models read 32,000. A chunk longer than the limit is searched by its first part only.
- **Price and where it runs.** The calculator compares every hosted model.

- Tool: [RAG and embedding cost](https://tecyfy.com/prices/embedding-cost). What a RAG system costs a month

2 details catch people. Many models want questions marked differently from passages: bge puts the line `Represent this sentence for searching relevant passages:` before the question only, and Voyage takes `input_type="query"` or `"document"`. And the choice locks you in: vectors from 2 models live in different spaces and cannot be compared, so changing models means embedding everything again. Keep the chunk text, not only the vectors, so that rerun is a script.

## Cut documents into chunks

You do not embed whole documents. One vector for a 6-page help centre has to stand for delivery, refunds and warranty at once, and matches every question a little. Cut each page into [chunks](https://tecyfy.com/concepts/chunking) that each answer one kind of question, and the search can return the one that matters.

(An interactive demo goes here on the page: https://tecyfy.com/ai-engineering/embeddings-and-search)

Cut along the document's own structure when it has one. Help pages have headings, so one chunk per section keeps each answer whole. Put the page title and the section heading at the top of every chunk: a section that only says "Order by 2pm on a working day..." does not say it is about express delivery, and its vector will not either. Module 5 goes further and has a model write that context.

## Search by meaning

With the vectors normalised to length 1, cosine similarity is a dot product, and comparing a question with every chunk is one matrix product:

```python
question = EMBEDDER.encode(QUERY_PREFIX + "Can you send my order to France?", normalize_embeddings=True)
scores = vectors @ question                 # one score per chunk, vectors being every chunk embedded
best = scores.argsort()[::-1][:3]           # the 3 highest
```

This is exact search: it compares with every chunk. On the laptop this course was tested on, a million vectors of 384 numbers took about 25 milliseconds. Past that, or when every millisecond counts, an approximate index such as [HNSW](https://tecyfy.com/concepts/hnsw) finds nearly the same top results by visiting a small part of the vectors.

Semantic search is at its best when the customer's words differ from the page's. It is at its worst on exact strings. Ask the shop's help pages "My mixer stopped and shows E04" and bge-small ranks the page on error E07 above the page on E04, because both are pages about mixer errors and the code barely moves the vector.

## Search by keyword

Keyword search catches what embeddings blur. The standard method, BM25, scores a chunk for each word of the question it contains, with 3 adjustments: rare words count more than common ones, a word repeated in a chunk adds less each time, and long chunks are evened out against short ones. Most databases and search engines have it built in. It is short enough to write yourself, which is the best way to see what it does:

```python
idf = math.log((n - df + 0.5) / (df + 0.5) + 1)  # df: chunks containing the word, n: all chunks
score += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / average))  # tf: times in this chunk
```

On "shows E04" it puts the E04 page first, because only one chunk contains `e04`. Its weakness is the other side. Asked "How long until I get my money back?", it ranks the warranty page first, which matches "how long", and a page about the mixer's bowl second, which contains "until". The refund page never says "money".

## Use both: hybrid search

[Hybrid search](https://tecyfy.com/concepts/hybrid-search) runs both searches and merges the lists. Their scores are on different scales, so the standard merge ignores scores and uses positions. **Reciprocal rank fusion** gives each chunk 1 / (60 + its rank) in each list and adds them up. The 60 comes from the 2009 paper that proposed it, which found the exact value mattered little.

```python
fused = Counter()
for ranking in (by_meaning(question, 20), by_keyword(question, 20)):
    for rank, chunk in enumerate(ranking, start=1):
        fused[chunk["id"]] += 1 / (60 + rank)
```

Fusion is good at one thing: making sure the right chunk is somewhere in the top 20. It is not good at putting it first. On "shows E04", meaning ranks E07 first and keyword ranks E04 first, so they tie. On "money back", the keyword list's mistakes pull the warranty page to the top. Ordering the shortlist is a separate job.

## Rerank the shortlist

A [reranker](https://tecyfy.com/concepts/reranking) reads the question and each candidate together and scores how well that chunk answers it. That is the accurate, slow kind of model the embeddings avoided, and it is affordable now because it reads 20 chunks, not all of them. Anthropic's tests reranked 150 candidates down to 20 and cut failed searches further than hybrid search alone.

```python
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
scores = reranker.predict([(question, f"{c['title']}\n{c['text']}") for c in candidates])
```

With a reranker after the fusion, the shop's search puts E04 first for "shows E04" and the refund page first for "money back". It is not free: a model runs on every question, which adds time. Module 5 measures whether each step earns its place.

## Where the vectors live

For a few thousand chunks, a NumPy array in memory is the whole vector store: 3,000 vectors of 384 numbers take under 5 MB. Move to a [vector database](https://tecyfy.com/concepts/vector-database) when you need what an array cannot give: filters ("only pages for this country"), updating chunks while the app runs, an index shared between servers, or millions of vectors behind an approximate index. If you already run Postgres, the pgvector extension puts the vectors next to the rest of your data before you add a new service.

- Comparison: [pgvector vs Qdrant vs Chroma](https://tecyfy.com/ai-agents/pgvector-vs-qdrant-vs-chroma). Start with pgvector if you already run Postgres - the vectors sit in the same tables, transactions and backups as the rest of your data, and a filter is an…

- Tool: [Vector database size calculator](https://tecyfy.com/ai-agents/vector-database-size). Documents, articles, tickets or products in; the vectors they make and the memory and disk the index needs out.

## Build it: search the shop's help pages

This is the search that module 5's assistant answers from. It reads a folder of help pages, cuts them into one chunk per section, and searches 4 ways, so you can see what each one finds.

Install the library. It brings PyTorch with it, and downloads the 2 small models the first time it runs:

```bash
pip install sentence-transformers
```

Make a folder called `docs` beside `triage.py` and save these 6 pages in it. They are short on purpose; the same code works on hundreds.

```markdown
# Delivery

## Delivery times and costs
Standard delivery takes 3 to 5 working days and costs £4.95. It is free on orders over £50.

## Express delivery
Order by 2pm on a working day and express delivery arrives the next working day. It costs £9.95 and is not available for large appliances.

## Where we deliver
We deliver to addresses in the UK and Ireland. We cannot send orders to other countries yet.

## Tracking an order
We email a tracking link when your order leaves our warehouse. If tracking has not changed for 3 working days, contact us with your order number and we will chase the courier.
```

That is `docs/delivery.md`. Then `docs/returns.md`:

```markdown
# Returns and refunds

## Returning an item
You can return any unused item within 30 days of delivery. Start a return from your account page and print the free returns label.

## Damaged or faulty on arrival
If an item arrives damaged, tell us within 48 hours and send a photo of the damage. A member of our team checks it and arranges a replacement or a refund.

## When your refund arrives
We refund to your original payment method within 5 working days of receiving the return. Your bank can take a few more days to show it.
```

`docs/warranty.md`:

```markdown
# Warranty

## What the warranty covers
Every product has a warranty against faults in materials and manufacture. It does not cover wear and tear, accidental damage or misuse.

## How long each product is covered
Electrical appliances are covered for 2 years. Stand mixers are covered for 5 years on the motor and 2 years on every other part. Knives have a lifetime warranty.

## Making a warranty claim
Send us your order number, a description of the fault and a photo or short video. If we confirm a fault, we repair or replace the product.
```

`docs/cookware.md`:

```markdown
# Pans and cookware

## Aster 26 cm frying pan (AST-26)
An aluminium frying pan with a non-stick coating. It works on every hob, including induction, and is oven safe up to 220°C once the silicone handle sleeve is removed. Wash it by hand.

## Aster 28 cm frying pan (AST-28)
The larger Aster pan, with the same non-stick coating and a helper handle. It works on every hob, including induction, and is oven safe up to 220°C. It can go in the dishwasher on a gentle cycle.

## Copper Classic 28 cm pan (CC-28)
A copper pan lined with stainless steel. It does not work on induction hobs. It is oven safe up to 260°C. Wash it by hand and polish the copper now and then.

## Caring for non-stick pans
Use wooden or silicone utensils and medium heat. Metal utensils and very high heat wear the non-stick coating faster.
```

`docs/knives.md`:

```markdown
# Knives

## Cleaning your knives
Wash knives by hand in warm soapy water and dry them straight away. Dishwashers can blunt the blade and leave spots on the steel.

## Sharpening
Hone the blade every few uses and sharpen it 2 or 3 times a year. We sharpen knives bought from us for free once a year.

## Lifetime warranty on knives
The lifetime warranty covers faults in the steel and the handle. It does not cover chips from cutting bones or damage from a dishwasher.
```

And `docs/mixer.md`:

```markdown
# Stand mixer help

## Error E01
The bowl is not locked in place. Turn the bowl clockwise until it clicks, then press start again.

## Error E04
The motor is too hot. Switch the mixer off at the wall and leave it for 30 minutes before you use it again. Heavy bread dough at high speed is the usual cause.

## Error E07
The speed sensor has a fault. Stop using the mixer and contact us with your order number, because this fault is covered by the warranty.

## Attaching the tools
The whisk, beater and dough hook push onto the drive shaft and turn a quarter turn to lock. Lift the head before you change a tool.
```

Now the search. Save it as `search.py`:

```python
"""Search the shop's help pages by meaning, by keyword, and both, then rerank."""
import math
import re
import sys
from collections import Counter
from pathlib import Path

from sentence_transformers import CrossEncoder, SentenceTransformer

EMBEDDER = SentenceTransformer("BAAI/bge-small-en-v1.5")
RERANKER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
# bge models search better when the question, and only the question, starts with this line.
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "


def chunk(path: Path) -> list[dict]:
    """One chunk per ## section, its page title and heading in front, so it says what it is about."""
    page, chunks = "", []
    for block in re.split(r"^## ", path.read_text(), flags=re.MULTILINE):
        if block.startswith("# "):
            page = block[2:].splitlines()[0].strip()
            continue
        heading, _, body = block.partition("\n")
        slug = re.sub(r"[^a-z0-9]+", "-", heading.lower()).strip("-")
        chunks.append({"id": f"{path.stem}#{slug}", "title": f"{page}: {heading.strip()}",
                       "text": body.strip(), "source": f"{path.stem}.md"})
    return chunks


def words(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())


class Index:
    def __init__(self, folder: str = "docs"):
        self.chunks = [c for path in sorted(Path(folder).glob("*.md")) for c in chunk(path)]
        texts = [f"{c['title']}\n{c['text']}" for c in self.chunks]
        self.vectors = EMBEDDER.encode(texts, normalize_embeddings=True)
        self.terms = [Counter(words(text)) for text in texts]
        self.lengths = [sum(terms.values()) for terms in self.terms]
        self.average = sum(self.lengths) / len(self.lengths)
        self.containing = Counter(word for terms in self.terms for word in terms)

    def by_meaning(self, query: str, k: int) -> list[dict]:
        q = EMBEDDER.encode(QUERY_PREFIX + query, normalize_embeddings=True)
        scores = self.vectors @ q  # the vectors have length 1, so this is cosine similarity
        return [self.chunks[i] for i in scores.argsort()[::-1][:k]]

    def by_keyword(self, query: str, k: int, k1: float = 1.2, b: float = 0.75) -> list[dict]:
        """BM25: rare words count for more, repeats count for less, long chunks are evened out."""
        n, scores = len(self.chunks), []
        for terms, length in zip(self.terms, self.lengths):
            score = 0.0
            for word in set(words(query)):
                if word in terms:
                    df = self.containing[word]
                    idf = math.log((n - df + 0.5) / (df + 0.5) + 1)
                    tf = terms[word]
                    score += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length / self.average))
            scores.append(score)
        ranked = sorted(range(n), key=lambda i: scores[i], reverse=True)
        return [self.chunks[i] for i in ranked[:k] if scores[i] > 0]

    def hybrid(self, query: str, k: int, pool: int = 20) -> list[dict]:
        """Reciprocal rank fusion: a chunk scores 1 / (60 + its rank) in each list it appears in."""
        fused, by_id = Counter(), {}
        for ranking in (self.by_meaning(query, pool), self.by_keyword(query, pool)):
            for rank, c in enumerate(ranking, start=1):
                fused[c["id"]] += 1 / (60 + rank)
                by_id[c["id"]] = c
        return [by_id[i] for i, _ in fused.most_common(k)]

    def search(self, query: str, k: int = 3, pool: int = 20) -> list[dict]:
        """Hybrid search for a pool of candidates, then a cross-encoder reads each one with the query."""
        candidates = self.hybrid(query, pool)
        scores = RERANKER.predict([(query, f"{c['title']}\n{c['text']}") for c in candidates])
        ranked = sorted(zip(scores, range(len(candidates))), reverse=True)
        return [candidates[i] for _, i in ranked[:k]]


if __name__ == "__main__":
    index = Index()
    query = " ".join(sys.argv[1:])
    for name in ("by_meaning", "by_keyword", "hybrid", "search"):
        found = getattr(index, name)(query, 3)
        print(f"{name:>10}: {', '.join(c['id'] for c in found) or 'nothing'}")
```

`Index` embeds every chunk when it starts, which takes a few seconds for these pages. For a real help centre you would embed once, save the vectors with the chunk ids, and embed again only the pages that change.

Try the 2 questions from this module:

```bash
python search.py "My mixer stopped and shows E04"
python search.py "How long until I get my money back?"
```

When we ran it, the first printed:

```text
by_meaning: mixer#error-e07, mixer#error-e04, mixer#error-e01
by_keyword: mixer#error-e04, mixer#error-e07, mixer#error-e01
    hybrid: mixer#error-e07, mixer#error-e04, mixer#error-e01
    search: mixer#error-e04, mixer#error-e07, mixer#error-e01
```

Meaning search puts E07 first, keyword search gets E04, the fusion ties them, and the reranker gets it right. The second question turns it around: keyword search starts with the warranty page and meaning search finds the refund page. Try your own questions, especially ones in the customer's words rather than the page's, and watch where each method goes wrong. `index.search()` is the one module 5 uses, and module 5 replaces "try a few questions" with a test that scores every method on the same list.

## Check yourself

1. A customer asks your shop's assistant "my mixer shows E04". Your vector search returns the page about error E07 first and E04 second. What is the change most likely to fix questions like this one?

   A. Switch to an embedding model with more dimensions
   B. Cut the help pages into smaller chunks
   C. Run BM25 keyword search beside the vector search and rerank the merged shortlist
   D. Send the top 10 passages to the model instead of the top 3

2. Your search uses bge-small-en-v1.5 and holds 40,000 embedded chunks. You switch to Voyage AI's voyage-4 for better results. What must you do with the 40,000 stored vectors?

   A. Embed all 40,000 chunks again with voyage-4, because vectors from 2 different models cannot be compared
   B. Nothing, since both models put similar texts close together
   C. Embed only new chunks with voyage-4 and keep the old vectors for the rest
   D. Normalise the old vectors to length 1 so they match voyage-4's

3. Your help centre has 3,000 chunks of 384 numbers each and gets a few questions a minute. A colleague wants to set up a managed vector database first. What do you recommend?

   A. The database, because vector search needs an approximate index to work at all
   B. Embed fewer chunks so the search stays fast
   C. Store the vectors as JSON in the chunk files and parse them on each question
   D. Start with the vectors in one array in memory, compared with one matrix product, and move to a database when you need filters, live updates or many more vectors; if you already run Postgres, pgvector is the first one to try

### Answers

1. C. Embeddings place both error pages close to the question, because both are about mixer errors, and a code like E04 barely changes the numbers. Keyword search matches E04 exactly, and a reranker reading the question with each page puts E04 first. A bigger model may still blur codes, the pages are already one error each, and sending 10 passages hides the problem instead of fixing the order.
2. A. Each model builds its own space of meaning, with its own length and its own directions, so a voyage-4 question vector compared with bge vectors gives meaningless scores. Every chunk is embedded again, and the index rebuilt. Keeping the chunk text, not only the vectors, is what makes that rerun cheap.
3. D. Comparing a question with 3,000 vectors of 384 numbers is one small matrix product, a fraction of a millisecond, and the array takes under 5 MB. An approximate index earns its place at millions of vectors, and a database when you need to filter by metadata, update chunks while serving, or share the index between machines. Parsing JSON on every question is slow for no gain.

## Questions

### What is an embedding in AI?

A list of numbers that an embedding model produces for a piece of text, an image or other data, such that things with similar meaning get similar lists. Comparing 2 lists, usually with cosine similarity, tells you how close in meaning the texts are, which is how semantic search finds a passage that shares no words with the question.

### What is the difference between semantic search and keyword search?

Semantic search compares embeddings, so it finds passages that mean the same as the question in other words, such as "refund" for "money back". Keyword search, usually BM25, matches the words themselves, so it finds exact terms an embedding blurs, such as an order number, an error code or a product name. Most production systems run both.

### What is hybrid search?

Running a keyword search and a vector search for the same question and merging the 2 ranked lists, most often with reciprocal rank fusion, which scores each result 1 divided by (60 plus its rank) in every list it appears in. It finds both exact terms and loose wording, and is usually followed by a reranking step.

### Do I need a vector database for RAG?

Not to start. A few thousand chunks fit in one array in memory and are compared with one matrix product in well under a millisecond. Move to a vector database when you need filtering by metadata, updates while serving, a shared index or millions of vectors, and if you already run Postgres, try the pgvector extension before a new service.

### Which embedding model should I use?

Test 2 or 3 on your own questions and documents, since the published leaderboards do not use your text. Weigh quality on your questions, the vector length (which sets memory), the longest input, the price, and whether you need to run it yourself. Anthropic does not make an embedding model and its documentation points to Voyage AI; open models such as bge-small-en-v1.5 run free on a laptop.

## Sources

- [Embeddings](https://platform.claude.com/docs/en/build-with-claude/embeddings)
- [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
- [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)
- [ms-marco-MiniLM-L6-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2)
- [Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)
- [Okapi BM25](https://en.wikipedia.org/wiki/Okapi_BM25)