SentencePiece
A tokenizer system that learns subword pieces directly from raw text, including spaces, without requiring word splitting first.
SentencePiece is a subword tokenizer system that can learn directly from raw text, which makes it useful when spaces are unreliable boundaries or when the text spans many languages and writing styles.
At a glance
Released
November 2018
Authors
Taku Kudo, John Richardson
Optimizes
- Open Vocabulary Coverage
- Pretokenization Independence
- Language Independent tokenization
- Raw Text preprocessing
What It Is
SentencePiece is a tokenizer system for building subword vocabularies. Instead of assuming the text has already been split into words, it trains on the original character stream and learns reusable pieces from that raw input.Why It Exists
Many languages do not mark word boundaries with simple spaces, and even languages that do still contain punctuation, mixed scripts, and messy formatting. SentencePiece keeps the tokenizer closer to the raw text so one training pipeline can work across multilingual or whitespace-agnostic data without relying on a separate word-splitting step first.How It Works
SentencePiece normalizes the text, treats whitespace as part of the tokenization stream, and then learns a vocabulary of subword pieces from that raw text sequence. In practice, that means a leading space marker can be preserved as part of a piece, and the model can later reconstruct where boundaries were instead of pretending spaces never existed. The same framework can train BPE-style merges or unigram-style token pieces, but the core idea is the same: learn directly from sentences rather than from pre-split word lists.Raw textNormalize the character streamKeep whitespace as a visible symbolLearn reusable subword piecesEncode text with those learned piecesFinal token sequenceRaw text to Normalize the character streamNormalize the character stream to Keep whitespace as a visible symbolKeep whitespace as a visible symbol to Learn reusable subword piecesLearn reusable subword pieces to Encode text with those learned piecesEncode text with those learned pieces to Final token sequence
Math Or Compute Schema
The schema below expresses the basic contract: a learned SentencePiece tokenizer maps a raw text stream into a token sequence without requiring a separate word-splitting stage first.- Raw text sequence before word splitting.
- Learned SentencePiece tokenizer.
- Output token sequence.
Compared To Nearby Modules
SentencePiece sits close to BPE and WordPiece because all three build reusable subword pieces. The reader-facing difference is where they start: SentencePiece is designed to learn from raw text with explicit whitespace handling, while WordPiece more often assumes a word-like split upstream and BPE discussions often focus on repeated merges over smaller starting symbols.| Comparison dimension | BPE | WordPiece | SentencePiece |
|---|---|---|---|
| Starting view of text | Starts from small symbols and merges common neighbors | Starts from subword candidates and grows a vocabulary | Trains directly on raw text without fixed word boundaries |
| How new pieces are chosen | Usually picks the most frequent adjacent pair | Uses a scoring rule instead of raw pair frequency alone | Can learn pieces with BPE-style or unigram-style training |
| Handling of spaces and bytes | Often paired with byte-level handling in GPT-style tokenizers | Usually assumes a word-like pre-tokenization step first | Treats spaces as regular symbols so boundaries stay explicit |
| Main tradeoff | Simple and reusable, but learned pieces can look unnatural | Works well for many text models, but depends more on the upstream text split | Flexible across languages, but its learned pieces may differ more from reader intuition |