# What is a sentence embedding, and how is it different from a word embedding?

One list of numbers for the meaning of a whole sentence or passage, made so that texts which mean the same thing get lists close together. It is what semantic search compares.

- Page: https://tecyfy.com/concepts/sentence-embedding
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Text Embedding, Sentence Vector, Bi-Encoder, Sentence-BERT

A sentence embedding turns a whole piece of text, a sentence, a question or a paragraph, into one list of numbers. Texts that mean the same thing get lists that sit close together, so "How long until I get my money back?" lands near "We refund within 5 working days" and far from "Stand mixers have a 5-year warranty on the motor".

It is the kind of [Embedding](https://tecyfy.com/concepts/embedding) that search, clustering and [RAG](https://tecyfy.com/concepts/rag) run on. A search embeds every passage once, embeds each question as it arrives, and returns the passages whose lists are closest, usually measured by [cosine similarity](https://tecyfy.com/concepts/cosine-similarity).

## Words versus sentences

Older [word embeddings](https://tecyfy.com/concepts#word-embedding) gave each word one fixed list. To get a sentence you averaged its words, which loses the order and the context: "the dog bit the man" and "the man bit the dog" come out the same. A sentence embedding model reads the whole text at once, with a [Transformer](https://tecyfy.com/concepts/transformer), and is trained so that the final list stands for what the text means.

## Why they made search fast

Before sentence embeddings, the most accurate way to compare 2 texts was to give both to a model like BERT at once. That works for one pair, but comparing a question with a million passages means a million model runs. The Sentence-BERT paper in 2019 put a number on it: finding the most similar pair among 10,000 sentences took about 65 hours that way, and about 5 seconds when each sentence was embedded once and the embeddings compared.

That is the trade every search system makes. Embedding each text separately, a **bi-encoder**, is fast and can be done ahead of time. Reading 2 texts together, a **cross-encoder**, is more accurate and too slow for a whole library, so it is used afterwards on a shortlist, a step called [Reranking](https://tecyfy.com/concepts/reranking).

## Using one

Hosted models are an API call: Voyage AI, which Anthropic's documentation points to since Anthropic makes none, OpenAI and Google all sell them, and the [embedding cost tool](/prices/embedding-cost) compares what each charges. Open models run on your machine through the Sentence Transformers library:

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
vectors = model.encode(["We refund within 5 working days.", "Stand mixers have a 5-year motor warranty."],
                       normalize_embeddings=True)
question = model.encode("Represent this sentence for searching relevant passages: When do I get my money back?",
                        normalize_embeddings=True)
print(vectors @ question)  # the refund sentence scores higher
```

2 details matter for search. Many models expect the question and the passages to be marked differently: bge models take a line before the question only, and Voyage has an `input_type` of query or document. And the lists from 2 different models cannot be compared, so changing the model means embedding every passage again.

## Related

- [Embedding](https://tecyfy.com/concepts/embedding): A list of numbers that stands for the meaning of a piece of text, an image or other data. Things with similar meaning get similar numbers, so a computer can find them.
- [Word Embedding](https://tecyfy.com/concepts#word-embedding): An embedding specifically representing individual words, capturing semantic relationships.
- [Cosine Similarity](https://tecyfy.com/concepts/cosine-similarity): A measure of how closely 2 vectors point the same way, from 1 for the same direction to −1 for opposite; with embeddings, a higher score means closer in meaning.
- [Dense Passage Retrieval](https://tecyfy.com/concepts/dense-passage-retrieval): Finding passages by comparing embeddings, from one encoder for questions and one for passages, instead of matching words. The 2020 DPR paper showed it beating BM25 keyword search on open-domain question answering.
- [Reranking](https://tecyfy.com/concepts/reranking): A second search step that reads each candidate passage together with the question and scores how well it answers, so the best few of a fast first search go to the model in the right order.

## Sources

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
- [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)
- [Embeddings](https://platform.claude.com/docs/en/build-with-claude/embeddings)