Compressed Sparse Attention
A DeepSeek-V4 attention variant that mixes compression and sparse access so very long prompts stay tractable.
Dense long-context attention becomes too expensive when every query wants every detailed key and value. Compressed sparse attention lowers that cost by keeping a compact memory view and only paying for selected detailed access.
At a glance
Optimizes
- Kv Cache
- Long Context Inference
- Memory Bandwidth
Example models
What It Is
This is an attention design for extremely long contexts. The key idea is not only sparsity and not only compression. It is the combination: a compact representation for broad coverage plus selective detailed retrieval where it matters.Why It Exists
The module targets KV-cache size, memory bandwidth, and long-context attention cost.How It Works
Tokens are first summarized into a compressed memory path. The query can then use sparse detailed access rather than reopening the full dense history every time.Long token historyCompressed memory pathSparse detailed lookupLong token history to Compressed memory pathCompressed memory path to Sparse detailed lookup
Math Or Compute Schema
The sketch below emphasizes the two-stage idea: compact memory first, selective detailed attention second.- compressed keys and values that summarize broad context
- the compression step
- the current query at time step t
- the selected sparse detailed positions
- the sparse attention computation over compressed and selected detailed memory