KV Cache

A DeepSeek-V4 attention variant that mixes compression and sparse access so very long prompts stay tractable.

An attention variant that reduces key-value cache memory by sharing key-value heads across query groups.

A DeepSeek-V4 long-context attention path that relies on a more aggressively compressed memory view when full-detail history is too costly.

An attention variant that compresses key-value cache storage into a low-rank latent space while keeping distinct query heads.

An attention variant that shares one key-value head across all query heads to minimize key-value cache memory.

Saved key and value tensors from earlier tokens that let later generation steps reuse attention state instead of recomputing the full prefix.

Compressing the cached keys and values kept across autoregressive decode steps to reduce long-context serving memory.

How quickly weights, activations, and KV cache bytes can move between memory and compute during model serving, and why that movement can cap useful throughput.

The prompt-processing stage that reads the full input once, builds the first attention state, and sets up generation before any reply token appears.

The serving setup that separates prompt processing from token-by-token generation because the two stages stress hardware, memory, and latency in different ways.

A throughput-ceiling chart that shows when memory movement or arithmetic compute limits attainable performance, using arithmetic intensity to explain which bound is active.

A serving system that keeps an active decode set refilled as requests finish or become compatible, so accelerators spend less time waiting for a whole batch to turn over.

A serving system practice that turns a trained model plus its runtime stack into a live, versioned inference service on real hardware.

The serving runtime layer that takes model weights plus live requests and turns them into token outputs by coordinating kernels, scheduling, memory movement, and cache state on real hardware.

A serving system concern that covers the live state the runtime must keep or move while answering requests, including model weights, KV cache growth, allocator overhead, and memory bandwidth limits.

A serving system that spills part of the KV cache to disk so very long sessions remain workable without keeping every token state in expensive fast memory.

A serving technique that uses a cheaper draft path to propose several next tokens, then asks the main model to verify them so decode-heavy requests can finish with fewer expensive turns.

The repeated next-token stage that reuses the KV cache and turns inter-token latency into the main reader experience after prefill finishes.

The serving latency metric that measures the delay between streamed output tokens after generation has started.

The saved key and value tensors a decoder keeps from earlier tokens so later tokens can reuse attention state instead of recomputing the full prefix.

The serving tradeoff between aggregate completed work over time and the wait experienced by an individual request under concurrency.

The serving latency metric that measures how long a reader waits from request start until the first generated token appears.

Tokens per second (tokens/s, tok/s, or TPS) is a throughput rate: how many model tokens a serving system generates per second over the measurement window you choose.