Fine-tuning and LoRA — full fine-tuning vs. LoRA compared
Fine-tuning teaches an already-trained model a little more. LoRA is not its opposite but one way of doing it. See how full fine-tuning and LoRA differ and when to pick each.
The picture, labeled
-
Full fine-tuning
Robot arms move every slider on the console. All the weights of the model are trained again.
-
LoRA Low-Rank Adaptation
The console is locked under glass. Only the small panel plugged into its side is adjusted.
LoRA is one way to fine-tune
Fine-tuning means taking a that has already been trained and it a little more on your own . LoRA is not the opposite of fine-tuning. It is one of the ways to do it.
When people just say "fine-tuning", they usually mean full fine-tuning, which updates every . LoRA is a lightweight method that trains only a tiny part. Lightweight methods like this are called PEFT (parameter-efficient fine-tuning).
Full fine-tuning
Updates every weight of the model.
parameter-efficient fine-tuning
Trains only a tiny part and leaves the rest alone.
- LoRA
Adds two small and trains only those.
LoRA on a model, so it needs even less .
So the real choice is not "fine-tuning or LoRA" but "full fine-tuning or LoRA".
Two ways to fine-tune
Both start from the same pre-trained model. They differ in how many of its weights they touch.
Pre-trained model
-
Full fine-tuning
full fine-tuning
All the weights of the pre-trained model are trained again on the new data.
- What is trained
- For a 7B model, all 7,000,000,000 parameters change.
- What you get
- The result is a whole new model, the same size as the original.
-
LoRA
Low-Rank Adaptation
LoRA (Low-Rank Adaptation) freezes the original weights W, attaches two small matrices, A and B, next to them, and trains only those two.
- What is trained
- The trained parameters are usually only 0.1–1% of the original.
- What you get
- The result is a small file that is added to the original model.
The LoRA formula
A model keeps its weights in large tables of numbers called matrices. LoRA does not edit the table W. It adds the product of two small tables, B and A, on top of it.
W′ = W + ΔW = W + B·A
The size of each matrix
- W
- d rows × d columns. It is not changed.
- B
- d rows × r columns. It is trained.
- A
- r rows × d columns. It is trained.
- r
- r is called the rank, and it is far smaller than d. (r ≪ d)
LoRA rests on one assumption: the change in the weights that fine-tuning causes, ΔW, can be approximated well enough by a .
How much smaller it is, in numbers
Take one layer where d = 4,096 and r = 8.
Full fine-tuning
16,777,216
trained parameters
all of W: d × d = 4,096 × 4,096
LoRA
65,536
trained parameters
B and A: 2 × d × r = 2 × 4,096 × 8
About 1/256 of the full count
LoRA is usually applied only to some matrices in the layers, so across the whole model the trained are only 0.1–1% of the original.
Full fine-tuning and LoRA side by side
| Item | Full fine-tuning | LoRA |
|---|---|---|
| What is trained | All the weights | Only the added low-rank matrices |
| and training time | Large | Much smaller, and smaller still when the model is quantized with |
| Size of the result | The whole model, from a few to tens of GB | An adapter file, from a few MB to a few hundred MB |
| Handling several tasks | One full copy of the model per task | One base model, and you swap only the adapter |
| speed | Same as the original | Same as the original once the adapter is merged into W |
| Forgetting what it knew | High risk | Relatively low, because the original is frozen |
| Expressive power | Highest | Limited by the rank r |
is when a model loses what it used to know while it learns something new.
Which one to choose
Most real-world work is done with LoRA. Full fine-tuning is for the cases where LoRA is not enough.
When LoRA fits
Most real-world work- You want to give a large base model a tone of voice, the vocabulary of a field or a specific .
- Your is limited.
- You need to run several versions for different uses.
When full fine-tuning fits
When LoRA is not enough- You must change the behaviour a lot, as with a new language or field the base model hardly knows.
- You have plenty of data and computing resources.
- The model is very small.
A tiny model does not need LoRA
For a tiny model with only millions to tens of millions of parameters, training everything is already cheap, so LoRA brings almost no benefit. LoRA is a technique that pays off more the larger the model is.
Sources
This page is a plain-language summary of the papers and documents below, checked on 2026-11-10. Where it differs, the original is right.
- LoRA: Low-Rank Adaptation of Large Language Models (arXiv)(opens in a new tab)
- QLoRA: Efficient Finetuning of Quantized LLMs (arXiv)(opens in a new tab)
- Hugging Face — PEFT(opens in a new tab)
- Hugging Face — PEFT: Adapters(opens in a new tab)
- Hugging Face — PEFT: LoRA(opens in a new tab)
- Hugging Face — Transformers: Fine-tuning(opens in a new tab)