Byte Pair Encoding
A subword tokenizer that learns frequent text pieces by repeatedly merging common neighboring symbols.
Byte pair encoding (BPE) is a tokenizer that builds a vocabulary from reusable text pieces instead of whole words, which helps models cover rare spellings and new words without exploding vocabulary size.
At a glance
Released
June 2016
Authors
Rico Sennrich, Barry Haddow, Alexandra Birch
Optimizes
- Open Vocabulary Coverage
- Vocabulary Size Efficiency
Example models
What It Is
Byte pair encoding (BPE) is a subword tokenizer. Instead of treating every whole word as one fixed unit, it starts from smaller symbols and learns which neighboring pieces appear together often enough to be merged into a larger reusable token.Why It Exists
BPE balances two pressures at once: the vocabulary should stay small enough to manage, but the tokenizer should still cover messy real text. Reusing common pieces like `ing`, `tion`, or `low` lets one vocabulary handle many rare or unseen words.How It Works
BPE begins with very small symbols, then repeatedly merges the most frequent adjacent pair in the training text. A simple example starts with pieces like `l` `o` `w`, `l` `o` `w` `e` `r`, and `n` `e` `w` `e` `s` `t`. If `l` + `o` appears often, the tokenizer adds `lo`. If `lo` + `w` is still common, it adds `low`. After enough rounds, words such as `lower` can be tokenized as pieces like `low` + `er` instead of six separate characters.Raw text piecesStart from tiny symbols or bytesCount neighboring pairsChoose the most frequent pairMerge into one new subwordReuse learned pieces such as `low` and `er`Final token sequenceRaw text pieces to Start from tiny symbols or bytesStart from tiny symbols or bytes to Count neighboring pairsCount neighboring pairs to Choose the most frequent pairChoose the most frequent pair to Merge into one new subwordMerge into one new subword to Reuse learned pieces such as `low` and `er`Reuse learned pieces such as `low` and `er` to Final token sequence
Math Or Compute Schema
The notation below compresses one training step into a single rule: take the current inventory of pieces, find the most useful neighboring pair, and merge that pair into a new subword unit.- Current token-piece inventory after merge step t.
- Most useful neighboring pair chosen at step t.
- Updated inventory after that pair becomes one new subword piece.
- Merge-step index.
Compared To Nearby Modules
BPE is one of several common subword tokenizers. WordPiece uses a similar piece-building idea but chooses merges with a different scoring rule, while SentencePiece is often trained directly on raw text and can avoid pre-tokenization assumptions about spaces or words.| Comparison dimension | BPE | WordPiece | SentencePiece |
|---|---|---|---|
| Starting view of text | Starts from small symbols and merges common neighbors | Starts from subword candidates and grows a vocabulary | Trains directly on raw text without fixed word boundaries |
| How new pieces are chosen | Usually picks the most frequent adjacent pair | Uses a scoring rule instead of raw pair frequency alone | Can learn pieces with BPE-style or unigram-style training |
| Handling of spaces and bytes | Often paired with byte-level handling in GPT-style tokenizers | Usually assumes a word-like pre-tokenization step first | Treats spaces as regular symbols so boundaries stay explicit |
| Main tradeoff | Simple and reusable, but learned pieces can look unnatural | Works well for many text models, but depends more on the upstream text split | Flexible across languages, but its learned pieces may differ more from reader intuition |