# What is QLoRA, and how does it fine-tune a large model on 1 GPU?

LoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning.

- Page: https://tecyfy.com/concepts/qlora
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: Quantized LoRA, Quantized Low-Rank Adaptation

[LoRA](https://tecyfy.com/concepts/lora) cuts what fine-tuning trains, but the frozen base model still has to sit in GPU memory at 16 bits, 2 bytes a parameter. For a 65 billion parameter model that alone is far more than 1 GPU holds. QLoRA, from a 2023 paper by Tim Dettmers and colleagues at the University of Washington, stores the frozen model in 4 bits instead and trains the LoRA adapter on top of it.

## The result

The paper fine-tuned a 65B model on a single 48 GB GPU, where ordinary 16-bit fine-tuning needed more than 780 GB, and matched the quality of 16-bit fine-tuning on its tasks. That put fine-tuning a large open model within reach of 1 GPU.

## The 3 tricks

Storing weights in 4 bits loses precision, and the paper notes that earlier [Quantization](https://tecyfy.com/concepts/quantization) methods worked for running a model but broke down during training. QLoRA added 3 things to keep quality:

- **4-bit NormalFloat (NF4).** Trained weights are roughly normally distributed, so NF4 spaces its 16 possible values to match that shape. The paper shows it is information-theoretically optimal for normally distributed weights.
- **Double quantization.** A 4-bit format keeps a scale number for each small block of weights. QLoRA quantizes those scales too, saving about 0.37 bits per parameter, around 3 GB on a 65B model.
- **Paged optimizers.** Long examples cause sudden memory spikes during training. Paged optimizers move optimizer state to ordinary memory when the GPU runs short, using NVIDIA's unified memory, instead of crashing.

The paper also applied LoRA to every linear layer rather than a few, which the `"all-linear"` setting in Hugging Face's PEFT library copies.

## In practice

In Hugging Face's TRL library, QLoRA is a LoRA config plus a 4-bit quantization config passed to the same trainer. The adapter it produces is an ordinary LoRA adapter. Serve it on the base model at whatever precision you choose, and measure quality on your own [eval set](https://tecyfy.com/concepts/eval-set) at that precision, since the model you test should be the model you ship.

The paper's other finding is worth keeping: fine-tuning on a small, high-quality dataset gave state-of-the-art results. Care spent on the examples pays more than a bigger pile of them.

## Related

- [LoRA](https://tecyfy.com/concepts/lora): A way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model.
- [Quantization](https://tecyfy.com/concepts/quantization): Storing a model's numbers with less precision, such as 4 bits instead of 16, so it needs far less memory and can run on smaller hardware, at some cost in accuracy.
- [Fine-Tuning](https://tecyfy.com/concepts/fine-tuning): Training an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first.
- [Parameter-Efficient Fine-Tuning](https://tecyfy.com/concepts/parameter-efficient-fine-tuning): Fine-tuning that freezes a model's own weights and trains a small number of new ones, such as a LoRA adapter, so a task costs a fraction of the memory and storage of full fine-tuning.
- [4‑bit Quantization](https://tecyfy.com/concepts#4-bit-quantization): Representing weights with 4‑bit integers.

## Sources

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)
- [SFT Trainer](https://huggingface.co/docs/trl/main/en/sft_trainer)
- [LoRA](https://huggingface.co/docs/peft/main/en/developer_guides/lora)