What is parameter-efficient fine-tuning (PEFT), and when should you use it instead of full fine-tuning?

Full Fine-TuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first. updates every weight in a model. That needs memory for the weights, their gradients and the optimizer's state for each one, and it produces a full new copy of the model for every task. Parameter-efficient fine-tuning, PEFT for short, freezes the original weights and trains only a small set of new ones. The name is also that of Hugging Face's library for doing it.

Why it took over

The cost of full fine-tuning grows with the model. The LoRA paper's example is GPT-3 175B: each fine-tuned copy has 175 billion parameters. Training only a small addition changes the arithmetic:

• Memory to train. No gradients or optimizer state for frozen weights. LoRA cut GPU memory 3 times on GPT-3 175B, and QLoRAQLoRALoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning., by keeping the frozen weights in 4 bits, fine-tuned a 65B model in under 48 GB where 16-bit fine-tuning needed more than 780 GB.
• Storage. 100 tasks on 1 base model take about 354 GB as LoRA adapters, against 35 TB as full copies, in the LoRA paper's sums.
• Serving. 1 copy of the base model in memory can serve many tasks, each request choosing its adapter.

And on the models the paper tested, quality matched full fine-tuning or beat it.

The main kinds

• LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. adds 2 small matrices beside chosen layers and learns a low-rank correction. It can be merged into the weights, so it adds no delay when the model runs. It is the usual choice today.
• QLoRA is LoRA trained on a 4-bit copy of the base model, for when memory is the limit.
• AdaptersAdapterA small bottleneck module inserted in each transformer layer to adapt a model to new tasks without updating full weights. insert small new layers into the network. They work, but the LoRA paper notes they add inference latency, since the extra layers run in sequence.
• Prompt and prefix tuning learn a few vectors placed in front of the input instead of changing any layer.

When to use it

Use PEFT when you fine-tune an open model at all: it is cheaper, the result is a small file you can version and roll back, and it rarely costs quality. Full fine-tuning is for when an adapter measurably falls short on your eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. and you have the hardware.

Training settings differ a little from full fine-tuning. Hugging Face's TRL notes that adapters usually take a higher learning rate, around 1e-4, because only the new weights are learning. The bigger question comes before any of this: whether the job needs fine-tuning at all, or a better prompt, a few examples or retrieval.