Heavily Compressed Attention
A DeepSeek-V4 long-context attention path that relies on a more aggressively compressed memory view when full-detail history is too costly.
Compressed sparse attention still keeps a route back to selected high-resolution detail. Heavily compressed attention goes further toward a compact memory view so very long contexts remain feasible even when detailed revisit must be rare.
At a glance
Optimizes
- Kv Cache
- Long Context Inference
- Memory Bandwidth
Example models
What It Is
Heavily compressed attention is the more aggressively compressed side of the DeepSeek-V4 hybrid attention story. Instead of preserving broad access to detailed tokens, it leans harder on a compact representation of old context.Why It Exists
The module targets extreme long-context efficiency, especially KV-cache growth and memory bandwidth pressure.How It Works
Older context is pushed into a more compressed memory structure, and queries interact with that compact summary rather than reopening a large detailed history by default.Long token historyHeavily compressed memoryCompact long-range queryLong token history to Heavily compressed memoryHeavily compressed memory to Compact long-range query