Cross-Attention
An attention pattern where queries come from one stream while keys and values come from a different memory source.
Cross-attention is the attention pattern where queries come from one stream while keys and values come from a separate memory source, so the active sequence can read information stored outside itself instead of only from its own hidden states.
At a glance
Released
June 2017
Authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
Optimizes
- Cross Source Conditioning
- Multistream Context Fusion
- Encoder Decoder Context Bridging
What It Is
Cross-attention is an attention variant in which the query vectors come from one stream, but the key and value vectors come from somewhere else. The model still performs a weighted lookup, but it no longer reads only from the same sequence that produced the query.Why It Exists
Self-attention is enough when one sequence only needs to mix information inside itself. Cross-attention solves the different problem of conditioning one stream on information stored elsewhere, such as a decoder reading an encoder memory or a text stack reading image features.How It Works
A target stream produces queries, while a separate source stream produces keys and values. Each query scores the external memory slots, turns those scores into weights with softmax, and blends the matching value vectors into a context vector for the target stream. The critical difference from self-attention is the memory source: the target tokens ask, but a different representation answers.q_tKV^X_0KV^X_1\cdotsKV^X_{s-3}KV^X_{s-2}KV^X_{s-1}q_t to KV^X_0q_t to KV^X_1q_t to \cdotsq_t to KV^X_{s-3}q_t to KV^X_{s-2}q_t to KV^X_{s-1}
Math Or Compute Schema
The formulas below contrast self-attention with cross-attention. The weighted lookup is still scaled dot-product attention, but cross-attention changes which hidden states create the queries versus the keys and values.- Hidden states from the same source sequence.
- Query projection of X.
- Key projection of X.
- Value projection of X.
- Key dimension per head.
- Hidden states from the external memory source.
- Hidden states from the active target stream.
- Query projection of Y.
- Key projection of X.
- Value projection of X.
- Key dimension per head.
Compared To Nearby Modules
Compared with self-attention, cross-attention changes where keys and values come from: self-attention builds queries, keys, and values from the same stream, while cross-attention keeps queries on the target stream but reads keys and values from a separate memory source. Compared with multi-head attention, the difference is not the memory layout: multi-head attention is the head-parallel execution pattern that splits attention across multiple heads and can run inside either self-attention or cross-attention blocks. Compared with causal attention, the key difference is masking and allowed positions, not the memory source: causal attention usually still reads the same growing sequence but blocks future positions, while cross-attention can read a different memory and may appear inside a causal decoder block. Compared with bidirectional attention, cross-attention again changes the memory source rather than simply opening left and right context inside one sequence.| Comparison dimension | Cross-Attention | Attention Overview | Multi-Head Attention | Causal Attention | Bidirectional Attention |
|---|---|---|---|---|---|
| Where queries come from | The active target stream that needs outside information | Depends on the attention pattern being discussed | Depends on the parent block: the same stream in self-attention or the target stream in cross-attention | The active generated prefix up to the current token | The visible sequence being encoded |
| Where keys and values come from | A different source sequence or memory bank | May be the same sequence or a separate memory depending on the design | Depends on the parent block: the same stream in self-attention or the external memory in cross-attention, split across parallel heads | The same growing sequence, but future positions are masked out | The same visible sequence with left and right context available |
| Main use context | Conditioning one stream on another, such as decoder-on-encoder or text-on-image | General weighted lookup across Transformer modules | Head-parallel attention execution that can wrap either self-attention or cross-attention lookups | Autoregressive decoding where each token predicts the next one without seeing future tokens | Encoder-style full-context understanding |