CLIP Image Tokenization

A vision tokenizer that turns fixed-size image patches into embedding vectors for transformer-style image encoders.

CLIP-style image tokenization divides an image into a grid of fixed-size patches, projects each patch into a vector token, and feeds that patch-token sequence into a transformer-style image encoder. These patch tokens are not text tokens; they are learned visual units that let vision models reuse the same attention machinery as language models.

At a glance

Released

February 2021

Authors

Alec Radford, Jong Wook Kim, Chris Hallacy, et al.

Optimizes

  • Fixed Grid Image Input
  • Shared Vision Transformer Input

What It Is

CLIP image tokenization is a vision tokenizer that converts a raw image into a sequence of patch tokens. The image is split into a regular grid of small squares, each square is flattened into a pixel vector, and a learned linear projection turns that vector into one embedding token. The resulting tokens are the discrete units a vision transformer consumes, similar in role to text tokens in a language model but built from local image regions instead of characters or subwords.

Why It Exists

Transformer blocks expect a sequence of vectors, not a raw height-by-width pixel grid. Image tokenization bridges that gap by turning local image structure into a fixed-length token stream the encoder can process with self-attention. A regular patch grid also keeps the input shape predictable, which simplifies batching, position handling, and reuse of standard transformer stacks across images of the same resolution.

How It Works

A typical CLIP-style vision encoder starts from a square image such as 224×224 pixels. A patch size of 16×16 yields a 14×14 grid, so the tokenizer produces 196 patch tokens before any optional class token is added. Each patch is flattened into a vector of raw pixel values, then multiplied by a learned projection matrix to produce one embedding per patch. Those embeddings form the token sequence passed into the image encoder. Position information is added separately through positional embeddings or another position mechanism, because the patch tokenizer itself mainly defines what each token represents, not where it sits in the grid.

Math Or Compute Schema

The two formulas below separate the tokenizer into its two core steps. First, the image becomes a grid of patches. Second, each patch vector is projected into one embedding token.
Patch grid from one image
P={p1,1,…,pH,W}P = \{p_{1,1}, \ldots, p_{H,W}\}
PP
Set of non-overlapping image patches covering the input.
HH
Number of patch rows after tiling the image.
WW
Number of patch columns after tiling the image.
pi,jp_{i,j}
Flattened pixel vector for the patch at grid row i and column j.
Linear projection of one patch
zi=Wpi+bz_i = W p_i + b
ziz_i
Embedding vector for patch i after projection.
WW
Learned projection matrix shared across patches.
pip_i
Flattened pixel vector for patch i.
bb
Learned bias vector added after projection.

Compared To Nearby Modules

Text tokenizers such as byte pair encoding (BPE) build tokens from learned text pieces or bytes. CLIP image tokenization instead builds tokens from fixed spatial patches in a pixel grid. Both produce a sequence the model can attend over, but the input alphabet, training objective, and failure modes differ. Image patch tokens inherit local visual context from their square region; text tokens inherit linguistic context from subword statistics.
Comparison dimensionCLIP Image TokenizationBPE
Input typeFixed-resolution imagesUnicode text strings
Base unitEqual-size pixel patches from a regular gridLearned subword pieces built from frequent symbol merges
Output formOne learned embedding vector per patchDiscrete token ids from a text vocabulary
Main tradeoffSimple and resolution-stable, but patch size locks local detail and sequence lengthFlexible open-vocabulary text coverage, but pieces are linguistic rather than spatial

Example Architectures

CLIP pairs this patch-token image encoder with a separate text encoder so image and caption embeddings can be aligned in one shared space. Vision Transformer (ViT) systems use the same patch-token pattern for image classification and later multimodal stacks extend it with cross-attention to text. Readers encounter the mechanism whenever a vision model advertises patch size, image resolution, or ViT-style encoding.

Limitations And Tradeoffs

Patch size fixes the local context each token can see. Smaller patches preserve finer detail but increase sequence length and compute. Larger patches shorten the sequence but blur fine structure. The tokenizer also assumes a fixed input resolution at training time, so changing image size later usually requires re-tiling, interpolation, or a different position scheme rather than a simple vocabulary swap.

Why It Still Matters

Patch-based image tokenization is the default front door for transformer vision encoders and many multimodal models built on top of them. Understanding how images become patch tokens explains why vision models report patch counts, why positional embeddings matter for grid order, and why image inputs are not interchangeable with text tokenizer outputs even when both are called tokens.

Tags

References

  1. Radford, Alec, et al. "Learning Transferable Visual Models From Natural Language Supervision." arXiv, 2021, https://arxiv.org/abs/2103.00020.
  2. Dosovitskiy, Alexey, et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." arXiv, 2020, https://arxiv.org/abs/2010.11929.

Byte-Level Tokenization

Previous Page

SentencePiece

Next Page

On this page

What It IsWhy It ExistsHow It WorksMath Or Compute SchemaCompared To Nearby ModulesExample ArchitecturesLimitations And TradeoffsWhy It Still MattersRelatedTagsReferences