What is a large language model?

ChatGPT, Claude and Gemini are all built on the same kind of program: a large language model, or LLM. Strip away the app and what is left does one job. It reads some text and predicts what comes next, one TokenTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens. at a time.

"Large" is about 2 things. The model is a very big set of numbers, called parameters or weights. And it learned those numbers from a very big pile of text. Meta's Llama 3, an open model from 2024, came in sizes of 8 billion and 70 billion parameters and was trained on over 15 trillion tokens.

How it learns

Training happens in stages.

Pretraining. The model is shown text with the next word hidden and asked to guess it. At first its guesses are close to random. After each guess its parameters are nudged a tiny amount toward the right answer, and this repeats over millions of sentences. Patterns of grammar, facts and style end up spread across the numbers. The model does not keep copies of the pages it read.

Post-training. A model that has only been pretrained is good at continuing text, not at answering questions or following instructions. So it is trained again, on examples of helpful answers and on people's judgements of which reply is better. This is what makes it an assistant.

Most of these models use the TransformerTransformerThe design behind almost every modern language model. It uses attention to let each word in a passage look at every other word, all at once. design, from a 2017 research paper, which handles a whole passage at once instead of word by word.

What follows from this

The model writes what is likely, not what it has checked. Most of the time likely and true agree. When they do not, you get a fluent, confident HallucinationHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist..

Its knowledge stops where its training text stopped, at its knowledge cutoffKnowledge CutoffThe date a model's training text ends. The model knows nothing after it unless it is given new text, for example by a web search., unless the app gives it search or your documents.

And it does not learn from your chat. Correct it and the correction lasts only as long as the conversation in front of it.

So treat an LLM as a very well-read writer with no notes in hand. Excellent at drafting, explaining and rewording. Worth checking whenever a fact, a number or a name matters.