# What is knowledge distillation, and how does a small model learn from a large one?

Training a small "student" model to copy the outputs of a large "teacher" model, so the small one keeps much of the large one's skill at a fraction of the cost to run.

- Page: https://tecyfy.com/concepts/knowledge-distillation
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Distillation, Model distillation, Teacher-student training

A large model is often much better than a small one, and much more expensive to run. Knowledge distillation trains the small model, the student, to reproduce what the large one, the teacher, does. The idea was named in a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean at Google.

## Why copying works better than the labels alone

A labelled example says only which answer is right. A teacher says more. The paper's example is a picture of a BMW: a good image model gives a garbage truck a very small chance, but still many times the chance it gives a carrot. Those small probabilities on wrong answers carry what the teacher has learned about how things relate. The paper calls them soft targets: the student is trained to match the teacher's whole spread of probabilities, not just the top answer, and learns more from each example than it would from the label.

## Distillation with language models

With language models the same idea often takes a simpler form. The teacher writes answers to a large set of inputs, and the student is [fine-tuned](https://tecyfy.com/concepts/fine-tuning) on those inputs and answers, often with [LoRA](https://tecyfy.com/concepts/lora). The aim, for a narrow job such as sorting support emails into categories, is a small open model that does that one job as well as the large one, at a fraction of the cost to run. Only your own eval can show whether it does.

The pattern has 3 parts:

1. **Inputs that look like real traffic.** Real emails, questions or documents, not made-up ones.
2. **Teacher answers you have checked.** The student learns the teacher's mistakes too, so review a sample, and drop or fix the wrong ones.
3. **The same [eval set](https://tecyfy.com/concepts/eval-set) for both.** Measure the student against the teacher on held-out examples before switching any traffic.

## Check the terms first

Using a hosted model's outputs to train another model can break that provider's terms. Anthropic's commercial terms, for example, forbid using the service to build a competing product, including training competing AI models. Read the terms of the model you use as the teacher before you start, or use an open model whose licence allows it.

## Related

- [Fine-Tuning](https://tecyfy.com/concepts/fine-tuning): Training an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.
- [Parameter-Efficient Fine-Tuning](https://tecyfy.com/concepts/parameter-efficient-fine-tuning): Fine-tuning that freezes a model's own weights and trains a small number of new ones, such as a LoRA adapter, so a task costs a fraction of the memory and storage of full fine-tuning.
- [LoRA](https://tecyfy.com/concepts/lora): A way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model.
- [Inference](https://tecyfy.com/concepts/inference): Using a trained model to produce an answer. Training builds the model once; inference is every time someone asks it something, and it is what you pay for per token.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.

## Sources

- [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)
- [Commercial Terms of Service](https://www.anthropic.com/legal/commercial-terms)