What is a transformer in AI?

Most of the language models you have heard of are built on one design, the transformer. It comes from a 2017 paper by 8 researchers, with a title that became famous: "Attention Is All You Need".

What came before

Earlier language systems read text the way you read a line of print, one word after another, carrying a running summary forward. That made them slow to train, because each step had to wait for the last.

What the transformer changed

The transformer drops the word-by-word reading. It looks at the whole passage at once and uses a mechanism called attentionSelf‑AttentionA mechanism that lets a model weigh the importance of different positions in a single sequence to decide, for each word, which other words matter to it.

Take "The trophy did not fit in the case because it was too big." To understand "it", the model has to connect it to "trophy", not "case". Attention is how it makes that connection. Each word asks a question of every other word, and the answers are weighted by how relevant they are. A transformer does this many times over, in many layers, and with several "heads" each looking for a different kind of link.

Because every word is handled at the same time, the work can be split across many chips. That is the practical reason the design won: in the paper it beat the older systems at translation while being far easier to run in parallel. Training on far more text, on far more chips, is what made models large.

The part that still runs one at a time

Reading is parallel. Writing is not. A language model built on a transformer still produces its answer one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. after another, and each new token can look back at everything before it but not ahead. The KV cacheKV CacheWork a model saves from the text it has already read, so it is not redone for every new token. Prompt caching keeps that work between requests and charges less for it. exists to make that step cheaper.

Why you might care

You never set up a transformer to use ChatGPT. But the design explains 2 things you do notice. A model can connect a detail on page 1 with a question on page 40. And the work of attention grows with the length of the text, faster than the text itself, which is part of why a long contextContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. is slow and costly.