CLIP

A dual-encoder multimodal model that learns to align text and images in one shared embedding space.

Contrastive Language-Image Pre-training (CLIP) learns to place captions and images near each other in the same vector space, which is why later image systems often reuse it to understand prompts instead of asking it to generate pixels directly.

At a glance

Family
clip
Source type
Research
Released
January 2021
Authors
Alec Radford, Jong Wook Kim, Chris Hallacy, et al.
Source
Learning Transferable Visual Models From Natural Language Supervision
Modalities
Text, Image, Multimodal
Parameter count
400 million parameters

What It Is

Contrastive Language-Image Pre-training (CLIP) is a multimodal model from OpenAI that learns from paired text and images. Instead of producing a paragraph or painting a picture, it learns representations: a text encoder turns a caption into a vector, an image encoder turns an image into a vector, and the model is trained so matching pairs land close together.

Inputs And Outputs

CLIP takes either text, images, or both. Its immediate outputs are embeddings and similarity scores rather than generated pixels. In practice, people use those outputs to rank images against text, retrieve matching examples, classify images with text labels, or supply conditioning signals to another model such as a diffusion pipeline.

Architecture

The core pattern is a dual encoder. One tower reads tokenized text, and the other reads image patches. Each tower maps its input into the same shared embedding space, then a similarity step compares the two vectors. During training, matched caption-image pairs are pulled closer together and mismatched pairs are pushed farther apart, so the shared space becomes useful for cross-modal comparison.

Important Modules

The modules below explain most of CLIP's mechanics. Bidirectional attention lets each encoder read its whole input, multi-head attention gives each tower several parallel comparison paths, and learned positional embeddings help preserve order inside token and patch sequences.

Training

CLIP's defining training idea is contrastive learning over large text-image datasets. For each minibatch, the model scores many captions against many images and learns to raise the score of the true pair while lowering the scores of the wrong pairings. That objective teaches alignment without asking the model to predict pixels or autoregressively emit text.

Practical Notes

CLIP became widely reused because it is good at turning prompts and images into comparable features. Retrieval systems use it for ranking, zero-shot classifiers use it for label matching, and text-guided image generation systems often use its text embeddings as conditioning. It is usually not the final image generator because it does not contain a decoder or denoiser that can synthesize pixels on its own.

Tags

References

  1. Radford, Alec, et al. "Learning Transferable Visual Models From Natural Language Supervision." arXiv, 2021, https://arxiv.org/abs/2103.00020.

BERT model

Previous Page

Cosmos 3

Next Page

On this page

What It IsInputs And OutputsArchitectureImportant ModulesTrainingPractical NotesRelated Models, Modules, And PapersTagsReferences