Tokenization

An encoder-only transformer that learns bidirectional text representations through masked language modeling pretraining.

Open model family from Google DeepMind with Gemma 4 as the current main line, spanning Effective 2B and 4B edge checkpoints through 12B, 26B mixture-of-experts, and 31B dense variants with native text, image, and audio input, thinking-mode reasoning, and Apache 2.0 licensing.

The smallest dense Qwen3 open-weight checkpoint: a 0.6-billion-parameter causal language model with 32,768-token context, Apache 2.0 licensing, and post-training for reasoning and instruction following.

An encoder-decoder transformer from Google Research that reformulates every NLP task as reading text and writing text in a shared token space.

A subword tokenizer that learns frequent text pieces by repeatedly merging common neighboring symbols.

A tokenizer design that starts from bytes so models can represent arbitrary text without unknown-character gaps.

A vision tokenizer that turns fixed-size image patches into embedding vectors for transformer-style image encoders.

A tokenizer system that learns subword pieces directly from raw text, including spaces, without requiring word splitting first.

A failure mode where text is split, labeled, or wrapped differently from what the model was trained or served to expect.

A subword tokenizer that keeps a candidate vocabulary and picks the highest-scoring whole segmentation instead of replaying merge rules.

A subword tokenizer that grows a vocabulary of reusable pieces and usually tokenizes each word with the longest matching subword sequence.

How tokenizers turn raw text into reusable pieces before embeddings, attention, and next-token prediction begin.

How images and video frames become patches, discrete codes, or latent tokens so sequence-based models can read visual data.

The paper that introduced deep bidirectional transformer encoders trained with masked language modeling and fine-tuned for language understanding tasks.

A technical report that introduced GPT-2 as a large decoder-only language model trained with broad next-token pretraining and byte-level BPE tokenization.

The large base-model training stage where a transformer learns broad language patterns from massive token sequences before later alignment or deployment shaping.

Reserved tokenizer markers such as beginning-of-sequence, end-of-sequence, padding, and separators that carry structural meaning instead of ordinary text meaning.