What is instruction tuning, and how does it turn a base model into an assistant?
Fine-tuning a pre-trained model on many tasks written as instructions with good answers, so it follows a request instead of just continuing the text.
beginner · Also called Instruction Fine-Tuning
A model fresh from pre-training is a text continuer. Give it "Write a short refund policy for a kitchen shop" and it was trained only to predict what text comes next, so it may treat the line as the start of a longer document and carry on writing that, rather than write the policy. Instruction tuning is the training step that teaches it to treat the text as a request and answer it.
How it is done
Take a pre-trained language modelLarge Language ModelA model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. and fine-tuneFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first. it on thousands of examples, each an instruction and a good response: "Translate this into French", "Summarise this article in 2 sentences", "Is this review positive or negative?". The model learns the general pattern of following an instruction, not only the tasks it saw.
That last point is what Google's FLAN paper showed in 2021. They tuned a 137 billion parameter model on over 60 tasks phrased as instructions and tested it on kinds of task it had never been tuned on. It beat the larger GPT-3, asked the same way without examples, on 20 of 25 of them.
Where it sits
Instruction tuning on written demonstrations is also called supervised fine-tuningSupervised Fine‑TuningStandard fine‑tuning on labeled input‑output pairs.. In OpenAI's InstructGPT work it was the first of 2 steps: tune on answers written by people, then improve further with RLHFReinforcement Learning from Human FeedbackA training step where people rank a model's answers, a reward model learns those preferences, and the language model is tuned to score well on it., where people rank the model's answers. Today's assistants all come out of a pipeline like this, which is why a model you call through an API answers questions and follows a system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say. rather than continuing your text.
For an engineer the practical point is that the model was trained to be helpful to whoever writes the instruction, and it sees every instruction in its context. That is useful when the instruction is yours and dangerous when it comes from a stranger's email.
Sources