Visual tokenization
How images and video frames become patches, discrete codes, or latent tokens so sequence-based models can read visual data.
Visual tokenization turns raw pixels from an image or video frame into a sequence of units a transformer, autoregressive, diffusion, or video model can process, and the main design choice is whether those units are spatial patches, discrete codebook symbols, or compressed latent features.
What It Is
Visual tokenization is the step that converts image or video-frame pixels into a sequence of model-readable units. Text models start from tokenizer output; visual models need an equivalent bridge because attention and autoregressive heads expect sequences, not raw raster grids. The output can be patch embeddings, discrete visual codes, or latent feature tokens, but the shared job is the same: turn a two- or three-dimensional visual field into an ordered list the rest of the stack can mix, predict, or denoise.Why It Matters
The representation choice shapes compute cost, memory use, and what kinds of generation are practical. Patch tokens keep local spatial structure visible to attention but grow with image resolution. Discrete codes compress visuals into symbol sequences that autoregressive generation can predict token by token. Latent tokens trade spatial detail for a smaller working space where diffusion models or transformer updates are cheaper than editing every pixel. Video adds time, so frame, tubelet, or latent temporal tokens decide how motion is packed into the same sequence machinery that image and video generation systems use.Main Representation Forms
Patch tokenization splits an image or frame into a grid of spatial regions and projects each region into an embedding that becomes one position in a sequence. CLIP image tokenization is a concrete patch-token example: it turns each image region into a sequence position that multimodal encoders can align with text. Discrete-code tokenization maps visual content into learned codebook IDs so autoregressive generation can predict the next visual symbol the same way it predicts text tokens. Latent-token approaches represent compressed image or video features that diffusion models and transformer systems can update more cheaply than raw pixels, often after an encoder shrinks the visual field into a lower-resolution latent grid. The comparison table below contrasts the three families by what visual data they start from, what each token looks like, where models typically use them, and the main tradeoff each choice introduces.| Comparison dimension | Patch tokens | Discrete visual codes | Latent visual tokens |
|---|---|---|---|
| Source data | Image or video-frame pixels divided into a spatial grid of regions | Image or frame content after encoder quantization into a learned codebook | Encoder-compressed image or video features rather than raw pixels |
| Token form | Continuous embedding vector per patch region | Integer codebook ID per visual unit, read as a symbol sequence | Lower-resolution latent feature map positions updated as tokens |
| Common model use | Vision transformers, CLIP-style image encoders, and multimodal understanding stacks | Autoregressive visual generation that predicts the next visual code like text tokens | Latent diffusion and denoising transformers that iterate on visual latents instead of full-resolution pixels |
| Main tradeoff | Keeps local spatial structure visible to attention, but sequence length grows with image resolution | Compresses visuals into a compact symbol stream, but reconstruction detail depends on codebook size and training | Makes iterative generation cheaper than pixel-space edits, but fine detail depends on the encoder and decoder pair |
Nearby Encoding And Generation Paths
Each representation family has a nearby page that shows the idea in a concrete system. Patch-token readers can follow CLIP image tokenization for a fixed-grid vision encoder. Discrete-code readers can follow autoregressive generation for symbol-by-symbol prediction. Latent-token readers can follow diffusion models for denoising over compressed visual features. Video readers can follow LTX-2.3 for frame and temporal tokenization in an audio-video diffusion transformer.Simple Example
Take one photograph of a street scene. CLIP image tokenization might cut the frame into a 16×16 grid of small squares, turn each square into one embedding vector, and feed a 256-position sequence to a vision transformer. A discrete-code model tied to autoregressive generation might instead encode the same scene into a shorter string of codebook IDs and predict the next ID symbol by symbol. A diffusion model working in latent space might compress the image into a coarse latent map and run many small denoising updates there instead of on full-resolution pixels.Common Confusions
Visual tokenization is not the same as text tokenization: both produce sequences, but visual methods must preserve spatial or compressed visual structure rather than word boundaries. Patch tokens are not the same as discrete visual codes; patches are continuous embeddings per region, while discrete codes are symbolic IDs from a learned vocabulary. Latent tokens are also not automatically smaller patch tokens; they usually come from a trained encoder that decides what detail to keep. Finally, one image can pass through more than one representation over a full system, such as patch encoding for understanding and latent tokens for generation.References
- Dosovitskiy, Alexey, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." arXiv, 2020, https://arxiv.org/abs/2010.11929.
- Radford, Alec, et al. "Learning Transferable Visual Models From Natural Language Supervision." arXiv, 2021, https://arxiv.org/abs/2103.00020.
- van den Oord, Aaron, Oriol Vinyals, and Koray Kavukcuoglu. "Neural Discrete Representation Learning." arXiv, 2017, https://arxiv.org/abs/1711.00937.
- Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-Resolution Image Synthesis with Latent Diffusion Models." arXiv, 2022, https://arxiv.org/abs/2112.10752.
- Lightricks LTX Team. "LTX-2: Efficient Joint Audio-Visual Foundation Model." arXiv, 2026, https://arxiv.org/abs/2601.03233.