What is in-context learning, and how is it different from fine-tuning?
A model doing a new task from instructions and examples in its prompt alone, with no change to its weights; it forgets the task when the call ends.
intermediate · Also called ICL
In-context learning is a model picking up a task from what is in its prompt. Show it 3 emails sorted into categories, then a fourth, and it sorts the fourth the same way. Nothing about the model changes: its weights are exactly as they were, and on the next call, without the examples, it has forgotten the task.
Why it was a surprise
Until about 2020 the way to teach a model a new task was Fine-TuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.: thousands of labelled examples and a training run. The GPT-3 paper showed that a large enough model could do many tasks from a description and a handful of examples given as plain text, with no training step at all, sometimes as well as fine-tuned models. It also listed tasks where this still fell short. That ability is the reason building on a model through an API is practical: you change its behaviour by changing the prompt.
What the model learns, and what it does not
"Learning" is a generous word. A 2022 study found that on classification tasks the examples' labels could be randomly wrong with little loss; what the model takes from them is the set of possible answers, the kind of input and the format. In other words, examples mostly tell the model what job this is and what an answer looks like. They are a weaker way to teach a precise rule than stating the rule, and a weaker way to teach new facts than giving the model the facts.
In-context learning or fine-tuning?
| In-context learning | Fine-tuning | |
|---|---|---|
| Changes the model | No | Yes, its weights |
| Cost to change | Edit the prompt | A training run |
| Cost per call | The examples are input tokens every time | No examples needed |
| Lasts | One call | Until you train again |
Start with the prompt: it is quicker to change and easy to test. Put facts that change, like prices and policies, in the context through retrievalRAGA way to answer from your own documents. A search finds the passages that match the question, and they are added to the prompt before the model writes. rather than training them in. Consider fine-tuning when the examples needed are too many or too costly to send with every call. Everything in context also competes for the model's attention: Anthropic notes that recall degrades as the contextContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out. grows, so more examples are not always better.
Sources