Sliding-Window Attention
An attention variant that restricts each query to a fixed local window of key positions instead of the full sequence.
Sliding-window attention is an attention variant that limits each token to a fixed local neighborhood, trading full-sequence reach for bounded compute and memory on long contexts.
At a glance
Released
April 2020
Authors
Iz Beltagy, Matthew E. Peters, Arman Cohan
Optimizes
- Attention Compute
- Memory Bandwidth
- Local Context Inference
What It Is
Sliding-window attention is an attention variant that limits each query position to keys within a fixed neighborhood. Instead of a full n-by-n attention matrix, only entries inside the window receive non-zero weight.Why It Exists
Sliding-window attention targets attention compute, memory bandwidth for score materialization, and the cost of serving long sequences when dense all-pairs attention would dominate runtime.How It Works
Each query position attends only to keys within W tokens to its left (and optionally right, depending on the pattern). Keys outside the window are zeroed before softmax normalization, so the model spends compute on local context rather than the full token grid.q_tKV_0KV_1\cdotsKV_{t-3}KV_{t-2}KV_{t-1}q_t to KV_{t-3}q_t to KV_{t-2}q_t to KV_{t-1}
Math Or Compute Schema
With sequence length n and window size W, sliding-window attention applies a band mask before softmax. The formulas below contrast dense multi-head attention against window-limited attention that restricts which keys each query can reach.Compared To Nearby Modules
Compared with multi-head attention, sliding-window attention limits reach to a local neighborhood rather than scoring all positions. Multi-query and grouped-query attention remain dense over allowed positions but reduce key-value head count. Sliding-window attention instead changes how far each query can look.| Comparison dimension | Sliding-Window Attention | Multi-Head Attention | Multi-Query Attention | Grouped-Query Attention |
|---|---|---|---|---|
| Attention locality | Fixed local window of W tokens per query | Full sequence; every query sees every key | Full sequence with shared key-value heads | Full sequence with grouped key-value heads |
| Compute scaling with sequence length | Linear O(n·W) per head when W is fixed | Quadratic O(n²) per head | Quadratic O(n²) with fewer key-value projections | Quadratic O(n²) with reduced key-value head count |
| Global token reach | Distant tokens require stacked layers or companion patterns | Any position can attend to any other position | Any position can attend to any other position | Any position can attend to any other position |