What is few-shot prompting, and how many examples should a prompt include?

Few-shot prompting means showing the model a few finished examples of the job before giving it the real one. Instead of describing the answer you want, you show 3 emails and the triage you would have written for each, then the new email. "Few-shot" is the number of examples; zero-shot is none, only the instruction.

Where it came from

The idea became famous with GPT-3 in 2020. Its paper, titled "Language Models are Few-Shot Learners", showed that a large enough model could do translation, question answering and other tasks from a few examples in the prompt, with no training at all, sometimes matching models fine-tuned for the task. That ability is in-context learningIn-Context LearningA model doing a new task from instructions and examples in its prompt alone, with no change to its weights; it forgets the task when the call ends..

What the examples actually teach

A 2022 study by Min and colleagues found something surprising: on classification tasks, giving the examples random wrong labels barely hurt. What mattered was that the examples showed the possible labels, the kind of input to expect and the format of the answer. For an engineer that is the useful lesson. Examples are most powerful for format, tone and the set of valid answers, and a rule you need followed exactly should also be stated as a rule.

How to write them

• 3 to 5 examples is Anthropic's suggestion for Claude.
• Make them like the real input. Real customer emails, typos and all, not tidy ones you wrote.
• Make them differ. If every example is a late delivery, the model learns that everything is a late delivery. Cover the categories and the edge cases that matter.
• Mark them off. Wrap each in <example> tags so the model can tell examples from instructions and from the real input.
• Pick canonical cases, not every case. Anthropic advises against stuffing a long list of edge cases into a prompt. Choose a few that show the behaviour clearly.
• Never test on your examples. A test case that is also an example passes for the wrong reason. Keep them separate.

Examples cost tokens on every call. When the same examples go with every request, put them in the fixed part of the prompt so prompt cachingPrompt CachingLetting the API keep the processed start of a prompt for a few minutes, so later requests that begin the same way are cheaper and quicker. can make them cheap, and when you need hundreds of examples to get the behaviour right, that is a sign to consider fine-tuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first..