Fine-tune an LLM with LoRA on one GPU: what fits, and the steps

Yes, one GPU is enough to fine-tune a useful model, as long as you train a LoRA adapter instead of every weight. On a 24 GB card you can fine-tune an 8B model with LoRA at 16 bits, or a 32B model with QLoRA at 4 bits. Full fine-tuning of the 8B model alone would need about 147 GB.

The reason is what sits in GPU memory. Full fine-tuning keeps gradients and 2 optimiser values for every weight. LoRA freezes the model, so only the small adapter needs them; the frozen model just has to fit. How LoRA's rank and alpha work is in LoRA rank and alpha explained; this article is about fitting the job on one card and running it.

The memory sum

3 things fill the GPU during training:

1. The frozen model. 2 bytes per weight at 16 bits. With QLoRA, the linear layers are stored in 4-bit NormalFloat with double quantization, about 4.127 bits per weight; we count the embedding and output layers at 16 bits.
2. The adapter's training state. Hugging Face's accounting for mixed-precision training is 18 bytes per trained weight: 6 for the weights, 4 for gradients and 8 for Adam's 2 values. At rank 16 on every linear layer, Qwen3-8B's adapter has 43.6 million weights, so about 0.8 GB.
3. Activations. What the forward pass keeps for the backward pass. They grow with sequence length and batch size, and gradient checkpointing (on by default in TRL) trades extra compute for much less of them. They are the part to measure, not predict.

For Qwen3 models at rank 16, before activations:

Read it against your card and leave room for activations: a few GB at least. On 24 GB, the 8B with LoRA (6.8 GB spare) and the 14B with QLoRA (12.9 GB spare) are comfortable; the 32B with QLoRA leaves 2.4 GB, so it needs short sequences and a batch of 1. On 80 GB, the 32B fits with LoRA at 16 bits.

Try itWill the fine-tune fit on your GPU?

Model

·Qwen3-4B·Qwen3-8B·Qwen3-14B·Qwen3-32B

Method

Full fine-tuneLoRA, 16-bitQLoRA, 4-bit

GPU memory

16 GB24 GB48 GB80 GB

LoRA rank

16

Model 6.1 GB Adapter training state 0.8 GB (43.6M weights)

Fits, with room for activations17.1 GB spare

The steps

1. Pick the model by the table, then by the task. For a narrow job (sorting tickets, pulling fields from documents, a fixed writing style), start small: a fine-tuned 4B or 8B model may be enough, and your test set will say whether it is. Our GPU memory calculator gives the size of any open model.

2. Write the data as prompt and completion pairs. One JSON object a line, the input you will really see and the exact answer you want. With this format, TRL learns from the completion only, so the model is taught to answer, not to repeat the prompt. Quality beats quantity: in the QLoRA paper, 9,000 well-chosen examples trained a better chatbot than 450,000. Keep 10% to 20% aside as a test set the model never sees.

{"prompt": "Ticket: My order KX-40211 never arrived.\nCategory:", "completion": " delivery"}

3. Train. The whole QLoRA run, with Hugging Face TRL and PEFT:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen3-8B"
quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quant, device_map="auto")

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

trainer = SFTTrainer(
    model=model,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM"),
    args=SFTConfig(
        output_dir="qwen3-8b-tickets",
        learning_rate=1e-4,              # adapters train at about 1e-4, not the 2e-5 default
        per_device_train_batch_size=2,   # the default is 8; lower it before anything else
        gradient_accumulation_steps=8,   # 2 x 8 = an effective batch of 16
        num_train_epochs=2,
        max_length=1024,
    ),
)
trainer.train()
trainer.save_model("qwen3-8b-tickets")
print(f"Peak GPU memory: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB")

For LoRA at 16 bits, leave out quant and load the model with dtype=torch.bfloat16. If the run fails with out of memory, change things in this order: batch size down (raise gradient accumulation to keep the effective batch), max_length down, then QLoRA, then a smaller model.

4. Test against the base model. Run the held-out examples through the base model and the adapter, and count right answers. If the adapter is not clearly better, the data needs work before the settings do.

5. Serve it. Merge the adapter into the weights for one fast model, or keep it separate and serve many adapters on one copy of the base model with vLLM (--enable-lora --lora-modules tickets=./qwen3-8b-tickets); each request names its adapter in the model field. The adapter file itself is small: 87 MB at 16 bits for the 8B model at rank 16.

What to do

1. Find your card's column in the table, then pick the largest model that leaves a few GB spare.
2. Spend the first day on 300 to 1,000 clean examples and a test set, not on settings.
3. Start from rank 16, alpha 32, all linear layers and a learning rate of 1e-4, and change one thing at a time.
4. Print the peak memory on every run, so the next model size is a sum, not a guess.

The ideas behind it are in LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. and QLoRAQLoRALoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning., and our free course builds a fine-tuned model end to end in its open models module.