What is QLoRA, and how does it fine-tune a large model on 1 GPU?
LoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning.
advanced · Also called Quantized LoRA, Quantized Low-Rank Adaptation
LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. cuts what fine-tuning trains, but the frozen base model still has to sit in GPU memory at 16 bits, 2 bytes a parameter. For a 65 billion parameter model that alone is far more than 1 GPU holds. QLoRA, from a 2023 paper by Tim Dettmers and colleagues at the University of Washington, stores the frozen model in 4 bits instead and trains the LoRA adapter on top of it.
The result
The paper fine-tuned a 65B model on a single 48 GB GPU, where ordinary 16-bit fine-tuning needed more than 780 GB, and matched the quality of 16-bit fine-tuning on its tasks. That put fine-tuning a large open model within reach of 1 GPU.
The 3 tricks
Storing weights in 4 bits loses precision, and the paper notes that earlier QuantizationQuantizationStoring a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy. methods worked for running a model but broke down during training. QLoRA added 3 things to keep quality:
- 4-bit NormalFloat (NF4). Trained weights are roughly normally distributed, so NF4 spaces its 16 possible values to match that shape. The paper shows it is information-theoretically optimal for normally distributed weights.
- Double quantization. A 4-bit format keeps a scale number for each small block of weights. QLoRA quantizes those scales too, saving about 0.37 bits per parameter, around 3 GB on a 65B model.
- Paged optimizers. Long examples cause sudden memory spikes during training. Paged optimizers move optimizer state to ordinary memory when the GPU runs short, using NVIDIA's unified memory, instead of crashing.
The paper also applied LoRA to every linear layer rather than a few, which the "all-linear" setting in Hugging Face's PEFT library copies.
In practice
In Hugging Face's TRL library, QLoRA is a LoRA config plus a 4-bit quantization config passed to the same trainer. The adapter it produces is an ordinary LoRA adapter. Serve it on the base model at whatever precision you choose, and measure quality on your own eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped. at that precision, since the model you test should be the model you ship.
The paper's other finding is worth keeping: fine-tuning on a small, high-quality dataset gave state-of-the-art results. Care spent on the examples pays more than a bigger pile of them.
Sources