BPE tokenizer — byte pair encoding merges the two pieces that appear together most often
BPE (Byte Pair Encoding) is a way to split text into tokens, the numbered pieces a model works with. It keeps merging the two pieces that appear together most often. This page explains why it is needed, how it is trained and used, and the words that come with it.
In a picture
-
Small pieces
The text is split into its smallest units.
-
A frequent pair
Find the two pieces that appear together most often.
-
A new piece
Merge that pair into one new piece.
What BPE is
BPE stands for Byte Pair Encoding. It is a way to split text into , the numbered pieces a model can work with.
It began in 1994 as a compression method. Today it is used to build . Every model in the family uses it.
There is one key idea: keep merging the two pieces that appear together most often.
It is like giving a short name to something you say often. People shorten a long phrase they use every day, and BPE does the same with letters.
Why it is needed
A model cannot read letters. It only works with numbers. There are three ways to turn text into numbers.
| Method | Good | Bad |
|---|---|---|
| By character | The is small and no word is unknown | The text becomes very long, so is slow |
| By word | The text becomes short | The vocabulary grows to millions of words, and a word never seen before cannot be handled |
| BPE (by piece) | In between the two. Frequent things stay whole and rare things are cut small | It has to be trained first |
- By character lower 5 pieces
- By word lower 1 pieces
- BPE (by piece) lower 2 pieces
In Korean
Korean has many word endings, as in 먹었습니다, 먹었어요, 먹었는데. That makes splitting by word especially weak. BPE learns by itself to split them like 먹었 + 습니다.
- 먹었습니다
- 먹었어요
- 먹었는데
same pieceending
How it is trained
A BPE tokenizer looks at a collection of training text (a corpus) and learns which pairs to merge. There are four steps.
-
Split
Split all the text into the smallest units ( or characters). This is the first vocabulary.
-
Count pairs
Count how many times each pair of neighbouring pieces appears.
-
Merge
Merge the most frequent pair into one new piece and add it to the vocabulary.
-
Repeat
Repeat steps 2 and 3 until the vocabulary reaches the size you want.
Repeat steps 2 and 3 until the vocabulary is big enough
Follow an example
Let us try it on the text "aaabdaaabac". We start with 11 characters.
-
Start
aaabdaaabac
11 pieces
-
"aa" is the most frequent pair (4 times). Merge it as Z = "aa".
ZabdZabac
9 pieces
-
"ab" is the most frequent pair (2 times). Merge it as Y = "ab".
ZYdZYac
7 pieces
-
"ZY" is the most frequent pair (2 times). Merge it as X = "ZY".
XdXac
5 pieces
The pair to merge nextA newly made piece
In step 2, "Za" also appears 2 times. When two pairs tie, each tokenizer follows its own rule to pick one. Here we picked "ab".
11 characters became 5 pieces. The order of the merges (aa → Z, ab → Y, ZY → X) is exactly what the tokenizer learned. The vocabulary started with 4 characters, and 3 merges made it 7.
How it is used
A trained tokenizer does two jobs.
Encoding
Split the new text into the smallest units. Then apply the merge rules learned in training, in the order they were learned, to get a list of token numbers.
Decoding
Turn the token numbers back into pieces and join them to get the original text.
- Text
baaab- Pieces
- bX
- Token numbers
[1, 6]- Text
baaab
Vocabulary (number : piece)0 : a1 : b2 : c3 : d4 : Z5 : Y6 : X
Encoding and then decoding must give back exactly the original text.
Terms
These words come up whenever BPE is explained.
Pieces and numbers
- Token
- The smallest piece of text a model works with. It can be one character, part of a word or a whole word.
- Token ID
- The number given to each token. This is the value that actually goes into the model.
- Vocabulary vocabulary, vocab
- The full list of tokens.
- Vocab size
- The number of starting units plus the number of merges. A bigger size makes the text shorter but makes the model's table bigger. For example, has 256 bytes plus 50,000 merges plus 1 , which makes 50,257.
Merging
- Pair
- Two tokens that sit side by side.
- Merge
- Joining one pair into one new token.
- Merge rules
- The list of merged pairs, in order. Saving a tokenizer means saving this list.
- Corpus
- The collection of text used to train the tokenizer.
Good to know
- Byte-level BPE
- BPE that starts from bytes (0 to 255) instead of characters. The first vocabulary is fixed at 256. Any character or emoji can be handled.
- Special token
- A token used as a signal, not as text. Examples are <eos>, which marks the end of the text, and <pad>, which fills the text to a fixed length.
- OOV out of vocabulary
- A word that is not in the vocabulary. This does not happen with byte-level BPE.
Sources
This page sums up the papers and documents below in simple words. They were checked on 2026-11-11. If they differ, the original is right.
- Neural Machine Translation of Rare Words with Subword Units (arXiv)(opens in a new tab)
- Language Models are Unsupervised Multitask Learners — GPT-2 (OpenAI)(opens in a new tab)
- Hugging Face — LLM Course: Byte-Pair Encoding tokenization(opens in a new tab)
- OpenAI — tiktoken(opens in a new tab)
- Wikipedia — Byte-pair encoding(opens in a new tab)