On-Disk KV Cache
A serving system that spills part of the KV cache to disk so very long sessions remain workable without keeping every token state in expensive fast memory.
Opening summary
Very long sessions can outgrow fast device memory even when the model weights fit. On-disk KV cache keeps the session alive by moving colder cache state to cheaper storage and pulling it back when needed.
At a glance
What It Is
This is a serving system, not a model module. It manages cached attention state once the active session becomes too large to hold entirely in the fastest memory tier.Where It Sits
It sits between the model's logical KV cache and the runtime memory hierarchy. The model still thinks it has cache. The system decides which parts stay hot and which parts spill.How It Works
Recent or valuable cache blocks remain hot, while colder blocks are spilled to slower storage. When later decoding needs them, the runtime reloads them instead of pretending the whole history stayed resident.On-Disk KV Cache System Flow
Prefill and cache buildSpill cold cacheReload on demandPrefill and cache build to Spill cold cacheSpill cold cache to Reload on demand
Cache writes and reuse
Cache reuse and reads