# What is LoRA, and how does low-rank adaptation fine-tune a large model cheaply?

A way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model.

- Page: https://tecyfy.com/concepts/lora
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: Low-Rank Adaptation, LoRA adapter

[Fine-tuning](https://tecyfy.com/concepts/fine-tuning) a model the usual way changes every one of its weights, so each task needs a full copy of the model and enough memory to train all of it. LoRA, short for low-rank adaptation, leaves the original weights frozen and learns a small correction beside them.

## How it works

A layer of a model multiplies its input by a large matrix of weights. LoRA adds 2 thin matrices next to it, `A` and `B`, whose product has the same shape as the big one but is built from far fewer numbers. Their width is the rank, `r`, a small number: PEFT's default is 8. During training only `A` and `B` change. The layer's output is the frozen weights' output plus the output of `B` times `A`, scaled by a setting called alpha divided by `r`.

`B` starts at zero, so on the first step the model behaves exactly like the original and the correction grows from there. The idea rests on an observation in the original 2021 paper from Microsoft: the change a task needs seems to have a low "intrinsic rank", so a thin correction can capture most of it.

## What it saves

On GPT-3 175B, the paper reports training memory falling from 1.2 TB to 350 GB, and the saved file for a task falling from 350 GB to 35 MB, about 10,000 times smaller, with the quality of full fine-tuning or better on the models it tested. The saving comes from not training, or storing optimizer state for, the frozen weights.

## Using it

- **Training.** Hugging Face's PEFT library wraps a model with a `LoraConfig`: the rank, alpha, which layers to adapt (`"all-linear"` adapts every linear layer) and dropout. A higher rank gives the adapter more capacity and more parameters to train.
- **Serving.** The adapter is a separate small file. You can merge it into the weights, so the model runs exactly as fast as before, or keep it separate and serve many adapters on 1 copy of the base model. [vLLM](https://tecyfy.com/concepts/vllm) does the second: each request names the adapter it wants.

LoRA changes how a model behaves, its format, labels and style. Like any fine-tuning, it is a poor way to add facts that change, which is what [RAG](https://tecyfy.com/concepts/rag) is for. [QLoRA](https://tecyfy.com/concepts/qlora) goes a step further and trains the adapter on a 4-bit copy of the base model.

## Related

- [Fine-Tuning](https://tecyfy.com/concepts/fine-tuning): Training an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.
- [Parameter-Efficient Fine-Tuning](https://tecyfy.com/concepts/parameter-efficient-fine-tuning): Fine-tuning that freezes a model's own weights and trains a small number of new ones, such as a LoRA adapter, so a task costs a fraction of the memory and storage of full fine-tuning.
- [QLoRA](https://tecyfy.com/concepts/qlora): LoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning.
- [Adapter](https://tecyfy.com/concepts#adapter): A small bottleneck module inserted in each transformer layer to adapt a model to new tasks without updating full weights.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.

## Sources

- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
- [LoRA](https://huggingface.co/docs/peft/main/en/developer_guides/lora)
- [Using LoRA for efficient fine-tuning](https://docs.vllm.ai/en/latest/features/lora/)