What is knowledge distillation, and how does a small model learn from a large one?

Training a small "student" model to copy the outputs of a large "teacher" model, so the small one keeps much of the large one's skill at a fraction of the cost to run.

intermediate · Also called Distillation, Model distillation, Teacher-student training

A large model is often much better than a small one, and much more expensive to run. Knowledge distillation trains the small model, the student, to reproduce what the large one, the teacher, does. The idea was named in a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean at Google.

Why copying works better than the labels alone

A labelled example says only which answer is right. A teacher says more. The paper's example is a picture of a BMW: a good image model gives a garbage truck a very small chance, but still many times the chance it gives a carrot. Those small probabilities on wrong answers carry what the teacher has learned about how things relate. The paper calls them soft targets: the student is trained to match the teacher's whole spread of probabilities, not just the top answer, and learns more from each example than it would from the label.

Distillation with language models

With language models the same idea often takes a simpler form. The teacher writes answers to a large set of inputs, and the student is fine-tuned on those inputs and answers, often with LoRA. The aim, for a narrow job such as sorting support emails into categories, is a small open model that does that one job as well as the large one, at a fraction of the cost to run. Only your own eval can show whether it does.

The pattern has 3 parts:

Inputs that look like real traffic. Real emails, questions or documents, not made-up ones.
Teacher answers you have checked. The student learns the teacher's mistakes too, so review a sample, and drop or fix the wrong ones.
The same eval set for both. Measure the student against the teacher on held-out examples before switching any traffic.

Check the terms first

Using a hosted model's outputs to train another model can break that provider's terms. Anthropic's commercial terms, for example, forbid using the service to build a competing product, including training competing AI models. Read the terms of the model you use as the teacher before you start, or use an open model whose licence allows it.

Sources

Where each fact comes from

Fine-Tuning
Training an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.
Parameter-Efficient Fine-Tuning
Fine-tuning that freezes a model's own weights and trains a small number of new ones, such as a LoRA adapter, so a task costs a fraction of the memory and storage of full fine-tuning.
LoRA
A way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model.
Inference
Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.
Quantization
Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.