Latent Diffusion Models

The research paper behind Stable Diffusion-style image generation: compress images into a latent space, denoise there instead of in pixel space, then decode the result.

Latent Diffusion Models (LDM) is the research paper behind Stable Diffusion-style image generation. The method learns a compressed latent representation space, runs iterative denoising inside that smaller space instead of on full-resolution pixels, and decodes the finished latent back into a visible image. Consumer-scale systems such as Stable Diffusion follow this recipe.

At a glance

Published

April 2022

Authors

Robin Rombach, Andreas Blattmann, Dominik Lorenz, et al.

Introduces

No introduced records listed yet.

Why It Matters

Pixel-space diffusion models treat every denoising step as a full-resolution image update. Each step must read and rewrite every pixel, so compute and memory grow sharply as width and height increase. Latent Diffusion Models showed that most perceptual structure can form inside a learned latent representation space that holds far fewer values than the pixel grid. Denoising there lowers compute and memory pressure while still delivering strong image quality at resolutions that would be impractical for pure pixel-space diffusion on typical hardware. That tradeoff is why Stable Diffusion-style pipelines compress first, denoise in latent space, and decode only at the end.

Method Or Architecture

The pipeline chains four stages. First, an autoencoder compresses an image into a learned latent tensor—a compact code in latent representation space rather than a pixel array. A diffusion model then runs its main generation loop as iterative denoising over that latent tensor: each step removes a little noise until a clean latent remains. Conditioning inputs such as text embeddings steer what the denoising steps should produce; Conditioning is the control signal that tells the generator which content or style to favor. After denoising completes, the autoencoder decoder maps the final latent back into pixels. The encoder and decoder are trained once; generation repeats only the latent denoising loop at inference time.
Generation flows from input image compression into latent denoising, with conditioning steering the loop, then decoding back to pixels.

Evidence

The paper demonstrates that latent-space denoising retains image quality while reducing compute and memory compared with pixel-space diffusion at similar output resolutions. The important evidence here is architectural: the authors train autoencoder compression, iterative denoising, and Conditioning as one coherent system and show the combination remains deployable where full pixel diffusion is not.

Limitations

Compression through an autoencoder can discard fine detail that pixel-space models might preserve, and reconstruction quality depends on encoder-decoder training. Weak Conditioning signals still produce weak guidance even when latent denoising is efficient. Readers comparing paradigms should weigh reconstruction fidelity against per-step cost rather than assuming latent denoising is strictly superior on every axis.

Tags

References

  1. Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-Resolution Image Synthesis with Latent Diffusion Models." arXiv, 2022, https://arxiv.org/abs/2112.10752.

GPT-2 Report

Previous Page

Learning Transferable Visual Models From Natural Language Supervision

Next Page

On this page

Why It MattersMethod Or ArchitectureEvidenceLimitationsRelated ToTagsReferences