Gated DeltaNet
A sequence-mixing module that updates a compact recurrent memory state with input-dependent gating and delta-rule writes instead of materializing a full attention matrix.
Gated Delta Networks, often shortened to Gated DeltaNet or GDN, are a sequence-mixing module that keeps a fixed-size recurrent memory state and updates it with gated decay plus targeted delta-rule writes rather than scoring every token pair in a full attention matrix.
At a glance
Released
December 2024
Authors
Songlin Yang, Jan Kautz, Ali Hatamizadeh
Optimizes
- Sequence Scaling
- Attention Compute
- Memory Bandwidth
- Long Context Inference
Example models
What It Is
Gated Delta Networks are a sequence-mixing module used in efficient long-context language models. Gated DeltaNet is the common shorthand for the same design. At each position the module projects the hidden state into query, key, and value signals, reads from a compact matrix-valued memory state, and writes back an updated state for the next step. The past is carried forward through that recurrent state instead of through an explicit all-pairs attention map.Why It Exists
Dense attention scales poorly as context length grows because every query token can compare against every earlier key. Linear attention lowers that cost by accumulating key-value statistics, but simple accumulation can blur or collide older information inside a fixed-size state. Gated Delta Networks exist to keep long-context sequence mixing affordable while giving the model stronger control over memory: gates can clear or retain past content, and the delta rule can replace a targeted slot instead of only adding another outer-product term.How It Works
Each step starts from the current hidden state and projects it into query, key, and value vectors. The gate α_t is a separate control path: it scales how much of the previous memory matrix S_{t-1} survives before any new content is written. The delta-rule path is different: it uses the current key and value to perform a targeted edit, replacing the memory content associated with k_t instead of only adding another outer-product term. After that gated write completes, the updated state S_t is stored for the next position and the current query q_t reads the result to form the layer output o_t. The diagram below keeps the gate decay step and the delta-rule write as distinct stages rather than one generic memory box.Gated DeltaNet compute path
Token hidden stateQ, K, V projectionsGate α_tDelta-rule writeUpdated memory S_tMemory S_{t-1}Output o_t = S_t q_tGating controls decay; the delta rule performs the targeted writeToken hidden state to Q, K, V projectionsQ, K, V projections to Gate α_tQ, K, V projections to Memory S_{t-1}Q, K, V projections to Delta-rule writeGate α_t to Delta-rule writeMemory S_{t-1} to Delta-rule writeDelta-rule write to Updated memory S_tUpdated memory S_t to Output o_t = S_t q_tQ, K, V projections to Output o_t = S_t q_t
Projections, memory, and readout
Gate decay control
Input or output step
Gating or memory stage
Mechanism note
Math Or Compute Schema
Dense multi-head attention materializes softmax weights over all earlier keys. Gated Delta Networks instead keep a matrix-valued memory state S_t. The gate α_t controls memory decay on the previous state, β_t scales a targeted delta-rule write built from v_t and the residual (v_t - S_{t-1} k_t), and the output o_t = S_t q_t reads the updated memory with the current query. The formulas below contrast the dense attention path with that gated delta recurrence.- Current-step query vector.
- Current-step key vector.
- Current-step value vector.
- Input-dependent gate that decays S_{t-1}.
- Input-dependent write strength for the delta-rule update.
- Matrix-valued memory state after gated decay and the delta write.
- Layer output read from S_t with q_t.
Compared To Nearby Modules
Multi-head attention keeps the full pairwise score matrix and remains the most direct way to address any earlier token. Linear attention compresses the past into a running summary, but ordinary accumulation-only updates can be harder to erase or replace precisely. DeltaNet-style updates use the same delta-rule write without the separate input-dependent gate that scales memory decay, so older content can linger unless another write overwrites it. Mamba2-style gating controls state decay in selective state-space mixers, but it does not use the same targeted key-value delta edit that Gated Delta Networks apply on top of their memory matrix. Sliding-window attention and sparse attention stay closer to dense softmax attention, yet they only connect tokens inside a local band or chosen pattern. Gated DeltaNet keeps one compact global recurrent state and updates it selectively at each step. That design trades direct token-by-token addressability for near-linear long-context cost: gated decay plus delta-rule editing can clear or replace memory slots without materializing an n-by-n attention map. Hybrid stacks that mix Gated DeltaNet with sliding-window attention or sparse attention are common when a model needs both efficient global mixing and direct local or patterned access.| Comparison dimension | Gated DeltaNet | Linear Attention | Multi-Head Attention | Sliding-Window Attention |
|---|---|---|---|---|
| Memory form | Fixed-size matrix-valued recurrent state carried across positions | Running key-value summary accumulated across positions | Explicit attention weights over all earlier tokens | Dense attention inside a fixed local window |
| Sequence scaling | Near-linear O(n) per layer with constant state width | Near-linear O(n) per head with fixed feature-map dimension | Quadratic O(n²) per head | Linear O(n·W) per head when window width W is fixed |
| Update style | Gated decay plus targeted delta-rule memory edits | Associative accumulation of feature-mapped keys and values | Softmax-weighted sum over all allowed key positions | Softmax over keys inside the local band only |
| Past addressability | Indirect; the past is compressed into one recurrent state rather than stored per token | Indirect; earlier tokens are summarized in accumulated statistics | Direct; any earlier token can be scored explicitly | Direct only inside the local window; distant tokens need other layers or patterns |