# What is self-attention in a transformer, and why does it matter for long prompts?

The step in a transformer where every token looks at the other tokens in the same text and takes in what is relevant to it, so meaning depends on context.

- Page: https://tecyfy.com/concepts/self-attention
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Intra-Attention, Scaled Dot-Product Attention

Self-attention is how a [Transformer](https://tecyfy.com/concepts/transformer) lets each word take its meaning from the words around it. In "the bank raised its rates" and "the boat reached the bank", the same token needs different meanings, and self-attention is the step that mixes in the context that decides which.

## How it works

Every [Token](https://tecyfy.com/concepts/token) is first turned into a list of numbers. For each token the model then makes 3 versions of it:

- a **query**: what this token is looking for,
- a **key**: what this token offers to others,
- a **value**: the information it passes on if chosen.

Each token's query is compared with every token's key. The closer the match, the larger the weight, and the token's new meaning is the weighted sum of all the values. In the 2017 paper that introduced the transformer this is called scaled dot-product attention, and "self" means the queries, keys and values all come from the same text.

A model does this many times in parallel ([attention heads](https://tecyfy.com/concepts#multi-head-attention)), each free to track something different, and stacks many such layers.

## Why an engineer should care

- **It is why a transformer trains fast.** Every position is computed at once instead of word after word, which the paper names as the reason it needs less training time than the recurrent networks before it.
- **It is why long prompts cost more.** Every token compares itself with every other, so the work grows with the square of the length. A [KV cache](https://tecyfy.com/concepts/kv-cache) saves the keys and values already computed so a model writing its answer does not redo them for every new token.
- **It only looks back.** In a model that writes text, each position may attend only to the positions before it. The model writes one token at a time, and each new token is fed back in as input for the next.
- **Everything in the [context window](https://tecyfy.com/concepts/context-window) can influence everything after it.** An instruction hidden in a pasted document sits in the same attention as your own, which is part of why [prompt injection](https://tecyfy.com/concepts/prompt-injection) works.

(An interactive demo goes with this on the page: https://tecyfy.com/concepts/self-attention)

## Related

- [Transformer](https://tecyfy.com/concepts/transformer): The design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once.
- [Attention Head](https://tecyfy.com/concepts#attention-head): One parallel attention mechanism within a multi‑head attention layer.
- [Multi‑Head Attention](https://tecyfy.com/concepts#multi-head-attention): Parallel attention layers allowing a model to focus on different subspaces
- [KV Cache](https://tecyfy.com/concepts/kv-cache): Work a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it.
- [Context Window](https://tecyfy.com/concepts/context-window): Everything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out.
- [Token](https://tecyfy.com/concepts/token): The small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.

## Sources

- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)