Bidirectional Attention
An attention pattern that lets each token read context on both sides when the model is allowed to see the full sequence.
Bidirectional attention lets a token use context from both earlier and later positions, which makes it the standard attention pattern for encoder-style models that need to understand a whole sequence instead of predicting only the next token.
At a glance
Released
June 2017
Authors
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.
Optimizes
- Full Sequence Context
- Representation Quality
What It Is
Bidirectional attention is the full-context version of self-attention. When the mask allows it, each token can compare itself with tokens to its left and right instead of looking in only one direction.Why It Exists
Bidirectional attention optimizes for representation quality when the model's job is to understand an entire sequence at once. It helps each position build a richer hidden state before any downstream prediction head reads it.How It Works
Queries, keys, and values are computed as in ordinary self-attention, but the attention mask does not block later positions. In a masked language model or other encoder setting, the score matrix stays open across the whole visible sequence, so one token can gather evidence from earlier words, later words, and its own position.q_t\cdotsKV_{t-2}KV_{t-1}KV_tKV_{t+1}KV_{t+2}\cdotsq_t to \cdotsq_t to KV_{t-2}q_t to KV_{t-1}q_t to KV_tq_t to KV_{t+1}q_t to KV_{t+2}q_t to \cdots
Math Or Compute Schema
The formulas below contrast causal attention with bidirectional attention. The projection math stays the same, but the mask changes from next-token-only reach to a full-context mask that keeps both left and right context available.Compared To Nearby Modules
Compared with causal attention, bidirectional attention removes the next-token-only restriction and uses full-sequence context. Compared with cross-attention, it still reads from the same sequence rather than a separate encoder memory. Compared with the broader attention page, this page focuses on the specific mask pattern that encoder-style models use.| Comparison dimension | Bidirectional Attention | Attention Overview | Multi-Head Attention |
|---|---|---|---|
| Visible context per query | Both earlier and later tokens in the same visible sequence | Depends on the architecture and mask pattern chosen for the attention block | Dense attention over whatever positions the chosen mask leaves visible |
| Best-fit task style | Encoder-style understanding and masked language model objectives | General attention lookup across many Transformer designs | Baseline per-head attention when the model keeps one key-value pair per query head |
| Where keys and values come from | The same sequence as the query position | Usually the same sequence for self-attention, but the overview spans broader patterns | The same sequence as the query position in self-attention blocks |