OkayStudy
EN English 한국어 日本語 中文 हिन्दी বাংলা اردو Español Português Français Deutsch Русский العربية Bahasa Indonesia Tiếng Việt ไทย Türkçe

How to enable notifications

Notifications are unavailable in this browser or on this address. Try a supported browser on the secure BaeumNova site.

If notifications are blocked, allow them in your browser settings.

  1. Open the site controls beside the address bar, or your browser’s site settings. Set Notifications for this site to Allow.
  2. In your device settings, allow notifications for your browser and check Do Not Disturb or Focus.
  3. Return to BaeumNova and reload the page. Once notifications are on, the bell disappears.

On iPhone or iPad, add BaeumNova to your Home Screen, open it there, then allow notifications.

BPE tokenizer — byte pair encoding merges the two pieces that appear together most often

BPE (Byte Pair Encoding) is a way to split text into tokens, the numbered pieces a model works with. It keeps merging the two pieces that appear together most often. This page explains why it is needed, how it is trained and used, and the words that come with it.

Back to the glossary Sources checked on 2026-11-11
Small blocks stand in a row on a long conveyor belt. In the middle a robot arm finds an orange block and a mint block that sit side by side and pushes them together. On the right a tray holds long blocks, each made of the two blocks joined into one

In a picture

  1. Small pieces

    The text is split into its smallest units.

  2. A frequent pair

    Find the two pieces that appear together most often.

  3. A new piece

    Merge that pair into one new piece.

What BPE is

BPE stands for Byte Pair Encoding. It is a way to split text into , the numbered pieces a model can work with.

It began in 1994 as a compression method. Today it is used to build . Every model in the family uses it.

There is one key idea: keep merging the two pieces that appear together most often.

It is like giving a short name to something you say often. People shorten a long phrase they use every day, and BPE does the same with letters.

Why it is needed

A model cannot read letters. It only works with numbers. There are three ways to turn text into numbers.

Three ways to turn text into numbers
Method Good Bad
By character The is small and no word is unknown The text becomes very long, so is slow
By word The text becomes short The vocabulary grows to millions of words, and a word never seen before cannot be handled
BPE (by piece) In between the two. Frequent things stay whole and rare things are cut small It has to be trained first
The word ⁦"lower"⁩ split in the three ways
  • By character lower 5 pieces
  • By word lower 1 pieces
  • BPE (by piece) lower 2 pieces

In Korean

Korean has many word endings, as in 먹었습니다, 먹었어요, 먹었는데. That makes splitting by word especially weak. BPE learns by itself to split them like 먹었 + 습니다.

The split BPE learned
  • 먹었습니다
  • 먹었어요
  • 먹었는데

same pieceending

How it is trained

A BPE tokenizer looks at a collection of training text (a corpus) and learns which pairs to merge. There are four steps.

  1. Split

    Split all the text into the smallest units ( or characters). This is the first vocabulary.

  2. Count pairs

    Count how many times each pair of neighbouring pieces appears.

  3. Merge

    Merge the most frequent pair into one new piece and add it to the vocabulary.

  4. Repeat

    Repeat steps 2 and 3 until the vocabulary reaches the size you want.

Repeat steps 2 and 3 until the vocabulary is big enough

Follow an example

Let us try it on the text ⁦"aaabdaaabac"⁩. We start with 11 characters.

  1. Start

    aaabdaaabac

    11 pieces

  2. ⁦"aa"⁩ is the most frequent pair (4 times). Merge it as ⁦Z⁩ = ⁦"aa"⁩.

    ZabdZabac

    9 pieces

  3. ⁦"ab"⁩ is the most frequent pair (2 times). Merge it as ⁦Y⁩ = ⁦"ab"⁩.

    ZYdZYac

    7 pieces

  4. ⁦"ZY"⁩ is the most frequent pair (2 times). Merge it as ⁦X⁩ = ⁦"ZY"⁩.

    XdXac

    5 pieces

The pair to merge nextA newly made piece

In step 2, ⁦"Za"⁩ also appears 2 times. When two pairs tie, each tokenizer follows its own rule to pick one. Here we picked ⁦"ab"⁩.

11 characters became 5 pieces. The order of the merges (⁦aa → Z, ab → Y, ZY → X⁩) is exactly what the tokenizer learned. The vocabulary started with 4 characters, and 3 merges made it 7.

How it is used

A trained tokenizer does two jobs.

Encoding

Split the new text into the smallest units. Then apply the merge rules learned in training, in the order they were learned, to get a list of token numbers.

Decoding

Turn the token numbers back into pieces and join them to get the original text.

Converting the new text ⁦"baaab"⁩ with the rules learned above
Text
baaab
Pieces
bX
Token numbers
[1, 6]
Text
baaab

Vocabulary (number : piece)0 : a1 : b2 : c3 : d4 : Z5 : Y6 : X

Encoding and then decoding must give back exactly the original text.

Terms

These words come up whenever BPE is explained.

Pieces and numbers

Token
The smallest piece of text a model works with. It can be one character, part of a word or a whole word.
Token ID
The number given to each token. This is the value that actually goes into the model.
Vocabulary vocabulary, vocab
The full list of tokens.
Vocab size
The number of starting units plus the number of merges. A bigger size makes the text shorter but makes the model's table bigger. For example, has 256 bytes plus 50,000 merges plus 1 , which makes 50,257.

Merging

Pair
Two tokens that sit side by side.
Merge
Joining one pair into one new token.
Merge rules
The list of merged pairs, in order. Saving a tokenizer means saving this list.
Corpus
The collection of text used to train the tokenizer.

Good to know

Byte-level BPE
BPE that starts from bytes (0 to 255) instead of characters. The first vocabulary is fixed at 256. Any character or emoji can be handled.
Special token
A token used as a signal, not as text. Examples are <eos>, which marks the end of the text, and <pad>, which fills the text to a fixed length.
OOV out of vocabulary
A word that is not in the vocabulary. This does not happen with byte-level BPE.

Sources

This page sums up the papers and documents below in simple words. They were checked on 2026-11-11. If they differ, the original is right.

Booking time has passed

Voice chat with Lyen Pick a language and a topic, then talk with Lyen by voice

Lyen speaks aloud in English · 한국어 · 日本語 · हिन्दी. In other languages she answers in text.

Voice chat with Lyen is for members. Sign up first, then start talking.