What is self-attention in a transformer, and why does it matter for long prompts?

Self-attention is how a TransformerTransformerThe design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once. lets each word take its meaning from the words around it. In "the bank raised its rates" and "the boat reached the bank", the same token needs different meanings, and self-attention is the step that mixes in the context that decides which.

How it works

Every TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. is first turned into a list of numbers. For each token the model then makes 3 versions of it:

• a query: what this token is looking for,
• a key: what this token offers to others,
• a value: the information it passes on if chosen.

Each token's query is compared with every token's key. The closer the match, the larger the weight, and the token's new meaning is the weighted sum of all the values. In the 2017 paper that introduced the transformer this is called scaled dot-product attention, and "self" means the queries, keys and values all come from the same text.

A model does this many times in parallel (attention headsMulti‑Head AttentionParallel attention layers allowing a model to focus on different subspaces), each free to track something different, and stacks many such layers.

Why an engineer should care

• It is why a transformer trains fast. Every position is computed at once instead of word after word, which the paper names as the reason it needs less training time than the recurrent networks before it.
• It is why long prompts cost more. Every token compares itself with every other, so the work grows with the square of the length. A KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. saves the keys and values already computed so a model writing its answer does not redo them for every new token.
• It only looks back. In a model that writes text, each position may attend only to the positions before it. The model writes one token at a time, and each new token is fed back in as input for the next.
• Everything in the context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. can influence everything after it. An instruction hidden in a pasted document sits in the same attention as your own, which is part of why prompt injectionPrompt InjectionAn attack that hides instructions in text a model reads, such as a web page, an email or a file, so the model follows the attacker instead of you. works.