Multi-Head Attention

The baseline attention design that gives every query head its own key-value head pair.

Multi-head attention (MHA) is the baseline transformer attention design, splitting attention into several parallel heads so different heads can focus on different relationships while each head keeps its own key and value pair.

At a glance

Released

June 2017

Authors

Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.

Optimizes

  • Expressiveness

What It Is

Multi-head attention (MHA) is the original scaled dot-product attention design used in the Transformer. Instead of running one large attention lookup, the model splits the work across several heads. Each head computes its own attention distribution over keys and values, and the head outputs are then merged.

Why It Exists

MHA optimizes for representational breadth. Distinct query heads can attend to different relationships in the same sequence without sharing key-value parameters.

How It Works

Linear projections produce multiple query heads, multiple key heads, and multiple value heads. Each query head attends to its matching key and value head, producing several outputs that are concatenated and projected again.

Math Or Compute Schema

The formula below is the per-head attention computation used inside multi-head attention. Each head index i pairs its own query, key, and value tensors before outputs are merged.
Multi-head attention (MHA)
Attention(Qi,Ki,Vi)=softmax ⁣(QiKi⊤dk)Vi\text{Attention}(Q_i, K_i, V_i) = \mathrm{softmax}\!\left(\frac{Q_i K_i^{\top}}{\sqrt{d_k}}\right) V_i
QQ
Query vectors for head i.
KK
Key vectors for head i.
VV
Value vectors for head i.
HH
Number of query heads.
dkd_k
Key dimension per head.
ii
Query head index.

Compared To Nearby Modules

Compared with multi-query attention, MHA stores one key-value pair per query head, so cache size grows with head count. Compared with grouped-query attention, MHA avoids shared key-value groups and keeps the largest cache footprint in the head-sharing family.
Comparison dimensionMulti-Head AttentionMulti-Query AttentionGrouped-Query Attention
Key-value head countH key heads and H value heads1 key head and 1 value headG key heads and G value heads
Query-head flexibilityH independent query and key-value head pairsH query heads share one key-value pairH distinct query heads grouped into G shared key-value pairs
Cache footprint per token2H tensors (full multi-head attention cache)2 tensors (single shared key-value cache)2G tensors (G keys plus G values)

Example Architectures

MHA appears in the original Transformer encoder-decoder, early GPT-style decoder-only language models, and many vision and speech transformers that adopt the standard attention block.

Limitations And Tradeoffs

Autoregressive inference must cache every key and value head, so memory and bandwidth grow with context length and head count. Serving long contexts often motivates multi-query attention or grouped-query attention instead.

Why It Still Matters

MHA is the conceptual baseline for every head-sharing variant. Understanding its one-to-one query-to-key-value mapping makes later tradeoffs easier to reason about.

Tags

References

  1. Vaswani, Ashish, et al. "Attention Is All You Need." arXiv, 2017, https://arxiv.org/abs/1706.03762.

Manifold-Constrained Hyper-Connections

Previous Page

Multi-Head Latent Attention

Next Page

On this page

What It IsWhy It ExistsHow It WorksMath Or Compute SchemaCompared To Nearby ModulesExample ArchitecturesLimitations And TradeoffsWhy It Still MattersRelatedTagsReferences