KV cache
The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.
What It Is
A KV cache is the running store of key and value tensors produced by earlier positions in a decoder. During the prefill stage, the model reads the prompt once and writes one cache entry per token per layer. During decode, each newly generated token appends one more entry, and attention reuses the saved keys and values instead of recomputing the whole prompt pass.Why It Matters
The cache is one of the main reasons long prompts and long replies cost real money to serve. Reusing saved attention state lowers latency because later steps avoid repeating prompt processing, but the cache also grows with sequence length, layer count, and key-value head layout. That means operators constantly balance time to first token, inter-token latency, and raw memory footprint when they choose head-sharing variants such as multi-query attention, grouped-query attention, or locality tricks such as sliding-window attention.Simple Example
Imagine a 2,000-token prompt. The first pass builds keys and values for those 2,000 positions. When the model generates token 2,001, attention reads the saved tensors for the prompt, computes fresh tensors only for the new position, and writes that extra entry back into the cache. Without the KV cache, the model would pay most of the prompt-processing cost again for every next token.Common Confusions
A KV cache is not the same as chat history saved in an app database: the product may store a transcript forever, while the model only keeps the active attention state for the current request. It is also not model weights or training memory. Finally, a smaller cache is not always a free win. MQA and GQA shrink the key-value state, and sliding-window attention limits how much old state stays active, but those designs change what information is preserved or how much detail each head can keep.Serving Path
KV cache sits between prompt processing and token-by-token generation. Follow the next two stages to see when the cache is first built and how later tokens keep reusing it.References
- Shazeer, Noam. "Fast Transformer Decoding: One Write-Head is All You Need." arXiv, 2019, https://arxiv.org/abs/1911.02150.
- Ainslie, Joshua, et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." arXiv, 2023, https://arxiv.org/abs/2305.13245.
- Liu, Zirui, et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." arXiv, 2024, https://arxiv.org/abs/2402.02750.