LoRA rank and alpha explained: what r and lora_alpha do, and what to set

In LoRA, the rank r sets how many weights the adapter trains, and lora_alpha sets how strongly the adapter's update is added to the model. Hugging Face PEFT multiplies the adapter's output by lora_alpha / r (or lora_alpha / √r with rsLoRA), so the ratio of the 2, not either number alone, decides the size of each step.

That is why "set alpha to twice the rank" works as a starting point: it keeps the scaling at 2 whatever rank you pick.

What LoRA trains

LoRALoRAA way to fine-tune a model by freezing its weights and training 2 small matrices beside each chosen layer, so a task needs a tiny file instead of a full copy of the model. freezes the model and puts 2 small matrices beside each chosen layer: A, which squeezes the layer's input down to r numbers, and B, which spreads them back out to the layer's output size. B starts at zero, so at the first step the model behaves exactly as before. The layer's output becomes:

output = W·x + (lora_alpha / r) · B·A·x

A layer that maps d_in inputs to d_out outputs gets r × (d_in + d_out) trainable weights. After training, the update can be merged into W, so the tuned model runs exactly as fast as the original.

Rank: how much the adapter can learn

The rank is the adapter's capacity. On Qwen3-8B (36 layers, hidden size 4,096, MLP size 12,288), the trainable weights are:

The LoRA paper found that a very low rank, even 1 or 2, was enough for GPT-3 on the tasks it tested, which suggests the change fine-tuning makes is often simple. Its authors do not expect that for every task: one far from the training data, such as a different language, needs more capacity. They cost more memory for gradients and optimiser state, though at these sizes that is small next to the model itself.

Which layers you adapt matters as much as the rank. PEFT's default adapts only the attention query and value layers; target_modules="all-linear" adapts every linear layer, as QLoRA did, and PEFT notes that this can match full fine-tuning.

Alpha: how loud the update is

lora_alpha is the volume knob. With the standard scaling:

• r = 16, lora_alpha = 32: update × 2
• r = 16, lora_alpha = 16: update × 1
• r = 64, lora_alpha = 32: update × 0.5

The LoRA authors point out that with the Adam optimiser, changing alpha is roughly the same as changing the learning rate. So they set alpha once, to the first rank they tried, and left it. That has a useful effect: when you later raise the rank, keeping lora_alpha fixed lowers the scaling, while keeping lora_alpha = 2r keeps it at 2 and your learning rate stays valid.

If training is unstable or the loss jumps, a lower scaling or learning rate helps. If the adapter barely changes the model's behaviour, the scaling may be too small, which is common at high ranks.

Try itWhat rank and alpha do to a LoRA adapter

Rank (r)

r = 16

48163264128256

lora_alpha

alpha = 32

q and vAttentionAll linear

alpha / rrsLoRA

Scaling2.00

Trainable share0.53%

LoraConfig(r=16, lora_alpha=32, target_modules="all-linear")

Usual range

rsLoRA: for higher ranks

Dividing by r means that at rank 256 the update is scaled down 16 times more than at rank 16. Kalajdzievski's rsLoRA paper shows this slows learning at high ranks, so raising the rank stops helping, and proves the right divisor is √r. In PEFT:

from peft import LoraConfig

config = LoraConfig(r=64, lora_alpha=32, use_rslora=True, target_modules="all-linear")

Here the scaling is 32 / √64 = 4, where standard LoRA would give 0.5. If you switch an existing recipe to rsLoRA, lower lora_alpha (or the learning rate) to keep the same strength; otherwise you have made every step 8 times larger.

Settings to start from

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

These are our starting values, not PEFT's defaults (PEFT ships r=8, lora_alpha=8). From there:

1. Measure first. Keep an evaluation set and change one setting at a time.
2. Not learning enough? Raise the rank (32, 64) with lora_alpha = 2r, or turn on rsLoRA for ranks of 64 and up.
3. Unstable or overfitting? Lower the learning rate or the scaling before lowering the rank.
4. Short of GPU memory? Load the base model in 4 bits: that is QLoRAQLoRALoRA trained on a copy of the base model stored in 4 bits, so fine-tuning a large model needs a fraction of the GPU memory, with 3 tricks that keep the quality of 16-bit fine-tuning., and the base model, not the adapter, is what fills the GPU. The GPU memory calculator shows what the base model needs.
5. Different layers need different settings? rank_pattern and alpha_pattern set them per layer.

What to do

• Think of lora_alpha / r as part of the learning rate, and keep the ratio fixed when you change the rank.
• Adapt all linear layers before reaching for a higher rank.
• Use rsLoRA when you go to rank 64 or more.
• Check print_trainable_parameters() matches what you expect from the table above.

For the wider picture of when to fine-tune at all, see Fine-TuningFine-TuningTraining an existing model a little further on your own examples so it picks up a task, a format or a style. Often a good prompt does the job first. and Parameter-Efficient Fine-TuningParameter-Efficient Fine-TuningFine-tuning that freezes a model's own weights and trains a small number of new ones, such as a LoRA adapter, so a task costs a fraction of the memory and storage of full fine-tuning., and how the Learning RateLearning RateThe size of the step a model's weights take on each training update. Too high and training blows up, too low and it barely moves. interacts with all of this.