WordPiece
A subword tokenizer that grows a vocabulary of reusable pieces and usually tokenizes each word with the longest matching subword sequence.
WordPiece is a subword tokenizer that breaks rare words into reusable pieces, which helps models cover new spellings without needing one vocabulary entry for every full word.
At a glance
Released
September 2016
Authors
Yonghui Wu, Mike Schuster, Zhifeng Chen, et al.
Optimizes
- Subword Coverage
- Vocabulary Efficiency
What It Is
WordPiece is a tokenizer that builds a vocabulary from subword pieces instead of only whole words. At inference time, it usually starts from a pre-tokenized word-like chunk and then chooses the longest matching pieces from its vocabulary until the chunk has been fully covered.Why It Exists
Whole-word vocabularies become large quickly and still fail on rare names, new product strings, or mixed-language text. WordPiece keeps the vocabulary smaller by reusing pieces such as prefixes, roots, and suffixes, while still letting common words stay whole when that is efficient.How It Works
Imagine the word "unhappiness". A WordPiece vocabulary might contain `un`, `happi`, and `##ness`. The tokenizer first splits the text into word-like chunks, then looks for the longest known piece at each step. If the whole word is not present, it falls back to smaller pieces until the full word can be represented. Compared with byte pair encoding (BPE), the reader-visible idea is similar, but WordPiece vocabulary building uses a scoring rule rather than choosing merges only by raw pair frequency.Word-like text chunkCheck for the longest vocabulary matchEmit the matched subword pieceRepeat on the remaining charactersFinal token sequenceFallback to a smaller piece when neededWord-like text chunk to Check for the longest vocabulary matchCheck for the longest vocabulary match to Emit the matched subword pieceCheck for the longest vocabulary match to Fallback to a smaller piece when neededEmit the matched subword piece to Repeat on the remaining charactersFallback to a smaller piece when needed to Repeat on the remaining charactersRepeat on the remaining characters to Final token sequence
Math Or Compute Schema
The notation below captures the practical contract: first break a word-like chunk into the longest reusable subword pieces that exist in the vocabulary, then emit those pieces as the token sequence for that chunk.Compared To Nearby Modules
WordPiece sits between BPE and SentencePiece in how people usually explain tokenizer families. Like BPE, it creates reusable subword units. Unlike SentencePiece, it is commonly taught with a word-like split upstream. That makes it a good bridge concept for readers moving from general tokenization into the tokenizer choices used by many encoder-style transformer models.| Comparison dimension | BPE | WordPiece | SentencePiece |
|---|---|---|---|
| Starting view of text | Starts from small symbols and merges common neighbors | Starts from subword candidates and grows a vocabulary | Trains directly on raw text without fixed word boundaries |
| How new pieces are chosen | Usually picks the most frequent adjacent pair | Uses a scoring rule instead of raw pair frequency alone | Can learn pieces with BPE-style or unigram-style training |
| Handling of spaces and bytes | Often paired with byte-level handling in GPT-style tokenizers | Usually assumes a word-like pre-tokenization step first | Treats spaces as regular symbols so boundaries stay explicit |
| Main tradeoff | Simple and reusable, but learned pieces can look unnatural | Works well for many text models, but depends more on the upstream text split | Flexible across languages, but its learned pieces may differ more from reader intuition |
Example Architectures
Readers most often encounter WordPiece in BERT-style model families and related encoder-heavy systems. Those models benefit from a tokenizer that can keep common words compact while still decomposing unfamiliar strings into meaningful smaller parts.Limitations And Tradeoffs
WordPiece still depends on upstream normalization and pre-tokenization choices, so the same sentence can break differently if punctuation, casing, or whitespace handling changes. Its pieces can also look less intuitive than full words, especially when one rare term becomes several short subword units.Why It Still Matters
WordPiece remains a useful reference because it shows one mainstream answer to the open-vocabulary problem. Once you understand how it reuses pieces and why it depends on a word-like split, it becomes easier to compare BPE, SentencePiece, and the token counts that shape model cost and context length.References
- Wu, Yonghui, et al. "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." arXiv, 26 Sept. 2016, arxiv.org/abs/1609.08144.
- Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, 2018, https://arxiv.org/abs/1810.04805.