Multi-Head Latent Attention

An attention variant that compresses key-value cache storage into a low-rank latent space while keeping distinct query heads.

Multi-head latent attention (MLA) is an attention variant that stores a compact latent key-value representation and reconstructs per-head keys and values when needed, reducing cache size without collapsing to one shared head.

At a glance

Released

May 2024

Authors

DeepSeek-AI

Optimizes

  • Kv Cache
  • Memory Bandwidth
  • Long Context Inference

What It Is

Multi-head latent attention (MLA) is an attention variant derived from multi-head attention. It keeps multiple query heads but projects keys and values into a shared low-rank latent space that is cached during autoregressive decoding.

Why It Exists

MLA targets key-value cache size, memory bandwidth during autoregressive decoding, and the cost of serving long contexts when full per-head key-value tensors would otherwise dominate memory.

How It Works

Keys and values are down-projected into a latent cache with rank r. During attention, those latent vectors are up-projected back to per-head key and value spaces. Queries remain head-specific, so each head attends against reconstructed key-value pairs derived from the shared latent cache.

Math Or Compute Schema

With multiple query heads and latent rank r, MLA caches compact latent vectors and reconstructs per-head keys and values through low-rank projections. The formulas below contrast full multi-head attention against MLA, which routes queries through a shared latent cache.
Multi-head attention (MHA)
Attention(Qi,Ki,Vi)=softmax ⁣(QiKi⊤dk)Vi\text{Attention}(Q_i, K_i, V_i) = \mathrm{softmax}\!\left(\frac{Q_i K_i^{\top}}{\sqrt{d_k}}\right) V_i
QQ
Query vectors for head i.
KK
Key vectors for head i.
VV
Value vectors for head i.
HH
Number of query heads.
dkd_k
Key dimension per head.
ii
Query head index.
Multi-head latent attention (MLA)
Attention(Qi,K^i,V^i)=softmax ⁣(QiK^i⊤dk)V^i,K^i=WiKc,  V^i=WiVc\text{Attention}(Q_i, \hat{K}_i, \hat{V}_i) = \mathrm{softmax}\!\left(\frac{Q_i \hat{K}_i^{\top}}{\sqrt{d_k}}\right) \hat{V}_i,\quad \hat{K}_i = W^K_i c,\; \hat{V}_i = W^V_i c
QQ
Query vectors for head i.
K^\hat{K}
Reconstructed key vectors from latent cache c.
V^\hat{V}
Reconstructed value vectors from latent cache c.
HH
Number of query heads.
rr
Latent rank of the compressed key-value cache.
dkd_k
Key dimension per head after up-projection.
ii
Query head index.
cc
Shared latent key-value cache vector stored per token.

Compared To Nearby Modules

Compared with multi-head attention, MLA stores a compressed latent cache instead of full per-head key-value tensors. Compared with multi-query and grouped-query attention, MLA keeps distinct query heads while compressing key-value state through low-rank projection rather than only sharing heads.
Comparison dimensionMulti-Head Latent AttentionMulti-Head AttentionMulti-Query AttentionGrouped-Query Attention
Key-value representationLow-rank latent key-value vectors (rank r)H key heads and H value heads1 key head and 1 value headG key heads and G value heads
Query-head flexibilityH distinct query heads with latent key-value reconstructionH independent query and key-value head pairsH query heads share one key-value pairH distinct query heads grouped into G shared key-value pairs
Cache footprint per tokenr-dimensional latent tensors (compressed key-value cache)2H tensors (full multi-head attention cache)2 tensors (single shared key-value cache)2G tensors (G keys plus G values)

Example Architectures

MLA appears in large decoder-only language models that prioritize efficient long-context inference while retaining multi-head query pathways.

Limitations And Tradeoffs

Low-rank key-value compression can limit representational flexibility versus storing full per-head tensors. The optimal latent rank depends on model scale and quality targets.

Why It Still Matters

As context windows grow, key-value cache efficiency remains a first-order serving constraint, so MLA offers a path between multi-head fidelity and aggressive cache compression through latent storage.

Tags

References

  1. DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv, 2024, https://arxiv.org/abs/2405.04434.

Multi-Head Attention

Previous Page

Multi-Query Attention

Next Page

On this page

What It IsWhy It ExistsHow It WorksMath Or Compute SchemaCompared To Nearby ModulesExample ArchitecturesLimitations And TradeoffsWhy It Still MattersRelatedTagsReferences