Causal Attention
An attention pattern that lets each token read earlier tokens and itself, but not later tokens, so the model can predict text one step at a time.
Causal attention is the decoder-style attention pattern that blocks future tokens. Each position can use the tokens that already exist in the prefix and its own position, but it cannot peek at tokens that have not been generated yet.
At a glance
Released
June 2017
Authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
Optimizes
- Autoregressive Generation
- Next Token Prediction
What It Is
Causal attention is the masked version of self-attention used for next-token prediction. For a query at position t, the mask keeps keys from positions 0 through t visible and blocks positions after t.Why It Exists
The mask preserves the training and inference rule that the model must predict the next token from only the prefix it has already seen. Without that restriction, a decoder could cheat by reading the answer from later positions.How It Works
Queries, keys, and values are computed as in ordinary self-attention, but the score matrix is multiplied by a causal mask before softmax. That mask keeps the diagonal and everything to the left, while zeroing scores for future positions. The result is an attention pattern where each token can read its earlier context and itself, but not tokens that come later in the sequence.q_tKV_0KV_1\cdotsKV_{t-3}KV_{t-2}KV_{t-1}q_t to KV_0q_t to KV_1q_t to \cdotsq_t to KV_{t-3}q_t to KV_{t-2}q_t to KV_{t-1}
Math Or Compute Schema
The formulas below contrast causal attention with bidirectional attention. The projections stay the same, but causal attention applies a triangular mask that blocks future positions, which is what makes token-by-token autoregressive generation possible.Compared To Nearby Modules
Compared with bidirectional attention, causal attention gives up right-context access so the model can produce text one token at a time. Compared with the broader attention page, this page focuses on the no-looking-ahead mask rather than the full query-key-value mechanism.| Comparison dimension | Causal Attention | Bidirectional Attention | Attention Overview |
|---|---|---|---|
| Visible context per query | Earlier tokens plus the current token only | Both earlier and later tokens in the same visible sequence | Depends on the architecture and mask pattern chosen for the attention block |
| Best-fit task style | Decoder-style next-token prediction and autoregressive generation | Encoder-style understanding and masked language model objectives | General attention lookup across many Transformer designs |
| Generation behavior | Produces output one token at a time without looking ahead | Not suitable for plain next-token decoding because future tokens stay visible | Varies with the chosen mask, sequence source, and objective |