Local Attention
A broad attention pattern that lets each query look only at a nearby neighborhood instead of the full sequence.
Local attention is a broad attention pattern that limits each token to nearby tokens, lowering long-context attention cost while making distant connections less direct.
At a glance
Released
April 2020
Authors
Iz Beltagy, Matthew E. Peters, Arman Cohan
Optimizes
- Attention Compute
- Memory Bandwidth
- Long Context Inference
What It Is
Local attention is a broad attention pattern where each query attends only to a nearby neighborhood instead of the full sequence. It is the umbrella idea behind window-limited attention layouts. Sliding-window attention is one common concrete version, while sparse attention is a broader family that may include local links plus other selective long-range links.Why It Exists
Dense attention compares every token with every other token, so score computation and score storage grow quadratically as context gets longer. Local attention keeps most of the useful nearby context while avoiding a full all-pairs score matrix on every layer.How It Works
For each query position, the model keeps only keys inside a local neighborhood and masks the rest before softmax. The neighborhood can be a fixed band, a block, or another nearby-only rule, but the core idea stays the same: spend attention compute on close tokens first instead of on the entire sequence.q_iKV_0KV_1\cdotsKV_{i-2}KV_{i-1}KV_iKV_{i+1}KV_{i+2}q_i to KV_{i-2}q_i to KV_{i-1}q_i to KV_iq_i to KV_{i+1}q_i to KV_{i+2}
Math Or Compute Schema
With sequence length n and local radius r, local attention applies a locality mask before softmax. The formulas below contrast dense multi-head attention against a masked local pattern that keeps only nearby token interactions.Compared To Nearby Modules
Compared with baseline attention, local attention changes where each query is allowed to look. Sliding-window attention is a specific fixed-width local-attention layout. Sparse attention is broader: it also removes many pairs, but it can keep arbitrary selected long-range links instead of only a neighborhood.| Comparison dimension | Local Attention | Attention | Sliding-Window Attention | Sparse Attention |
|---|---|---|---|---|
| Reach pattern | Broad umbrella for nearby-only neighborhoods | Full sequence; every query can score every key | Fixed-width band around each query position | Selected subset of token pairs, which may be local, blockwise, or mixed |
| Compute scaling with sequence length | Often near-linear O(n·r) when the local radius stays fixed | Quadratic O(n²) per head | Linear O(n·W) per head when W is fixed | Subquadratic when the allowed-pair fraction stays fixed |
| Distant-token path | Usually indirect; far information moves through layers or companion links | Direct one-hop access to any token | Indirect unless another pattern adds longer-range access | Depends on the sparsity pattern; some designs keep explicit long-range links |