Unigram Tokenizer
A subword tokenizer that keeps a candidate vocabulary and picks the highest-scoring whole segmentation instead of replaying merge rules.
A unigram tokenizer starts with many possible word pieces and chooses the highest-scoring full segmentation of a string, which is why it feels different from merge-based tokenizers such as byte-pair encoding (BPE).
At a glance
Released
August 2018
Authors
Taku Kudo, John Richardson
Optimizes
- Open Vocabulary text
- Language Independent preprocessing
What It Is
A unigram tokenizer is a subword tokenizer that treats segmentation as a scoring problem over whole candidate segmentations. Instead of constructing text by replaying one merge rule after another, it keeps a learned vocabulary of possible pieces and selects the segmentation that scores best for the input string.Why It Exists
Unigram tokenization exists so a tokenizer can keep a fixed-size subword vocabulary, still cover open-vocabulary text, and choose among multiple plausible segmentations instead of hard-coding one merge path. That makes it useful for rare words, spacing-heavy text, and multilingual raw-text pipelines such as SentencePiece-style training.How It Works
Training begins with a large candidate vocabulary of subword pieces. For a new string, the tokenizer considers several valid paths through those pieces, scores each path from the learned piece probabilities, and emits the best full segmentation before turning the pieces into token IDs.Raw textCandidate subword vocabularySeveral valid segmentationsHighest-scoring pathFinal token IDsRaw text to Candidate subword vocabularyCandidate subword vocabulary to Several valid segmentationsSeveral valid segmentations to Highest-scoring pathHighest-scoring path to Final token IDs
Math Or Compute Schema
The formulas below contrast unigram scoring against merge-based byte-pair encoding. Unigram chooses the best whole path under a learned piece distribution, while BPE deterministically replays merge operations that were fixed during tokenizer training.- Input text string.
- One candidate segmentation of the string.
- Set of valid segmentations for x under the candidate vocabulary.
- One subword piece inside a candidate segmentation.
- Learned score or probability assigned to piece p.
- Current segmentation after t merge steps.
- The merge rule chosen from the learned ordered merge list.
- Segmentation after applying merge rule m_t.
Compared To Nearby Modules
Compared with BPE, a unigram tokenizer does not ask which pair should merge next inside the string. It scores complete candidate segmentations instead. That usually makes the tokenizer behavior easier to describe as a probabilistic vocabulary model, while BPE remains easier to explain as a sequence of merges.| Comparison dimension | Unigram tokenizer | BPE |
|---|---|---|
| Training rule | Start with a large candidate vocabulary and prune pieces by score | Learn an ordered list of frequent merges |
| Inference rule | Choose the highest-scoring full segmentation of the string | Replay the learned merges to build pieces step by step |
| Reader mental model | A probabilistic vocabulary that picks the best path | A merge history that rewrites text into larger pieces |
Example Architectures
Readers most often meet unigram tokenization through SentencePiece-style training pipelines. It is common in systems that want language-independent preprocessing or that prefer learning from raw sentences rather than a fixed external pre-tokenizer.Limitations And Tradeoffs
A unigram tokenizer still commits to one final segmentation at inference time, so the learned vocabulary quality matters a lot. It can also feel less intuitive than BPE to debug by hand because the winning path depends on piece scores across the whole segmentation, not just on an easily listed merge history.Why It Still Matters
Tokenization decisions still shape context length, cost, and how cleanly a model handles multilingual or unusual text. Knowing the unigram approach helps readers recognize why two modern tokenizers can produce very different pieces even when both use subwords.References
- Kudo, Taku, and John Richardson. "SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2018, pp. 66-71, aclanthology.org/D18-2012/.
- Sennrich, Rico, Barry Haddow, and Alexandra Birch. "Neural Machine Translation of Rare Words with Subword Units." arXiv, 10 June 2016, arxiv.org/abs/1508.07909.