OkayStudy
EN English 한국어 日本語 中文 हिन्दी বাংলা اردو Español Português Français Deutsch Русский العربية Bahasa Indonesia Tiếng Việt ไทย Türkçe

How to enable notifications

Notifications are unavailable in this browser or on this address. Try a supported browser on the secure BaeumNova site.

If notifications are blocked, allow them in your browser settings.

  1. Open the site controls beside the address bar, or your browser’s site settings. Set Notifications for this site to Allow.
  2. In your device settings, allow notifications for your browser and check Do Not Disturb or Focus.
  3. Return to BaeumNova and reload the page. Once notifications are on, the bell disappears.

On iPhone or iPad, add BaeumNova to your Home Screen, open it there, then allow notifications.

Fine-tuning and LoRA — full fine-tuning vs. LoRA compared

Fine-tuning teaches an already-trained model a little more. LoRA is not its opposite but one way of doing it. See how full fine-tuning and LoRA differ and when to pick each.

Back to the glossary Sources checked on 2026-11-10
Two mixing consoles side by side: on the left, robot arms move every slider; on the right, the console is locked under a glass cover and one arm adjusts only a small panel plugged into its side

The picture, labeled

  1. Full fine-tuning

    Robot arms move every slider on the console. All the weights of the model are trained again.

  2. LoRA Low-Rank Adaptation

    The console is locked under glass. Only the small panel plugged into its side is adjusted.

LoRA is one way to fine-tune

Fine-tuning means taking a that has already been trained and it a little more on your own . LoRA is not the opposite of fine-tuning. It is one of the ways to do it.

When people just say "fine-tuning", they usually mean full fine-tuning, which updates every . LoRA is a lightweight method that trains only a tiny part. Lightweight methods like this are called PEFT (parameter-efficient fine-tuning).

Fine-tuning

Full fine-tuning

Updates every weight of the model.

parameter-efficient fine-tuning

Trains only a tiny part and leaves the rest alone.

  • LoRA

    Adds two small and trains only those.

  • LoRA on a model, so it needs even less .

So the real choice is not "fine-tuning or LoRA" but "full fine-tuning or LoRA".

Two ways to fine-tune

Both start from the same pre-trained model. They differ in how many of its weights they touch.

Pre-trained model

  1. Full fine-tuning

    full fine-tuning

    All the weights of the pre-trained model are trained again on the new data.

    What is trained
    For a 7B model, all 7,000,000,000 parameters change.
    What you get
    The result is a whole new model, the same size as the original.
  2. LoRA

    Low-Rank Adaptation

    LoRA (Low-Rank Adaptation) freezes the original weights W, attaches two small matrices, A and B, next to them, and trains only those two.

    What is trained
    The trained parameters are usually only 0.1–1% of the original.
    What you get
    The result is a small file that is added to the original model.

The LoRA formula

A model keeps its weights in large tables of numbers called matrices. LoRA does not edit the table W. It adds the product of two small tables, B and A, on top of it.

W′ = W + ΔW = W + B·A

W′weights used after tuning Woriginal weights frozen ΔWthe change
ΔWthe change Bsmall matrix trained Asmall matrix trained

The size of each matrix

W
d rows × d columns. It is not changed.
B
d rows × r columns. It is trained.
A
r rows × d columns. It is trained.
r
r is called the rank, and it is far smaller than d. (r ≪ d)

LoRA rests on one assumption: the change in the weights that fine-tuning causes, ΔW, can be approximated well enough by a .

How much smaller it is, in numbers

Take one layer where d = 4,096 and r = 8.

Full fine-tuning

16,777,216

trained parameters

all of W: d × d = 4,096 × 4,096

LoRA

65,536

trained parameters

B and A: 2 × d × r = 2 × 4,096 × 8

About 1/256 of the full count

Trained parameters in one layer (d = 4,096, r = 8)
  • Full fine-tuning 16,777,216
  • LoRA 65,536

Both bars use one scale, so the LoRA bar is almost a dot.

LoRA is usually applied only to some matrices in the layers, so across the whole model the trained are only 0.1–1% of the original.

Full fine-tuning and LoRA side by side

Full fine-tuning vs. LoRA
Item Full fine-tuning LoRA
What is trained All the weights Only the added low-rank matrices
and training time Large Much smaller, and smaller still when the model is quantized with
Size of the result The whole model, from a few to tens of GB An adapter file, from a few MB to a few hundred MB
Handling several tasks One full copy of the model per task One base model, and you swap only the adapter
speed Same as the original Same as the original once the adapter is merged into W
Forgetting what it knew High risk Relatively low, because the original is frozen
Expressive power Highest Limited by the rank r

is when a model loses what it used to know while it learns something new.

Which one to choose

Most real-world work is done with LoRA. Full fine-tuning is for the cases where LoRA is not enough.

When LoRA fits

Most real-world work
  • You want to give a large base model a tone of voice, the vocabulary of a field or a specific .
  • Your is limited.
  • You need to run several versions for different uses.

When full fine-tuning fits

When LoRA is not enough
  • You must change the behaviour a lot, as with a new language or field the base model hardly knows.
  • You have plenty of data and computing resources.
  • The model is very small.

A tiny model does not need LoRA

For a tiny model with only millions to tens of millions of parameters, training everything is already cheap, so LoRA brings almost no benefit. LoRA is a technique that pays off more the larger the model is.

Sources

This page is a plain-language summary of the papers and documents below, checked on 2026-11-10. Where it differs, the original is right.

Booking time has passed

Voice chat with Lyen Pick a language and a topic, then talk with Lyen by voice

Lyen speaks aloud in English · 한국어 · 日本語 · हिन्दी. In other languages she answers in text.

Voice chat with Lyen is for members. Sign up first, then start talking.