Memory
A serving system concern that covers the live state the runtime must keep or move while answering requests, including model weights, KV cache growth, allocator overhead, and memory bandwidth limits.
Opening summary
Serving memory is the live state an inference runtime must keep nearby or move quickly enough to answer requests. It is not just the advertised capacity number on a device spec sheet. What matters in practice is which weights stay resident, how the key-value cache grows as sessions stay active, and how fast the system can move that state without stalling the next token.
At a glance
Released
July 2022
System type
Memory
Related models
No related models listed yet.
Related modules
No related modules listed yet.
What It Is
Memory is a serving-system concern because a model can only answer requests when the runtime can hold or fetch the state needed for the next step. That live state includes resident model weights, the growing key-value cache, temporary activations, and the bookkeeping overhead required to place them on usable hardware.Where It Sits
It sits underneath visible serving choices such as batching, routing, deployment, and prefill versus decode scheduling. Those systems do not change the learned model architecture itself. They decide how work is placed so the right memory stays available, close enough, and fast enough for the inference engine to keep serving.How It Works
Before useful work begins, the runtime needs the model weights resident in a fast enough memory tier to execute the forward pass. Prefill usually creates the sharpest one-time burst of cache growth because the system has to read the whole prompt and write the first key-value cache entries for every token in it. Decode then reuses those entries and extends them token by token, so the pressure shifts from a big prompt-ingest burst to a longer-lived accumulation problem while active requests remain alive. That is why memory stays a live serving concern after load: the stack keeps balancing resident weights, cache growth, and free space for the next step instead of treating memory as a one-time loading problem. Memory capacity answers how much state can exist at once, while memory bandwidth answers how quickly the runtime can read or move that state. A system can have enough total bytes and still miss targets when the next layer cannot pull weights or active cache blocks fast enough. Fragmentation adds a second limit because enough memory on paper does not always mean enough usable memory in time.Memory System Flow
Keep model weights residentPrefill reads the whole promptKV cache gets its first burstLive serving memory keeps changingDecode reuses and extends cacheKeep model weights resident to Prefill reads the whole promptPrefill reads the whole prompt to KV cache gets its first burstPrefill reads the whole prompt to Decode reuses and extends cacheKV cache gets its first burst to Live serving memory keeps changingDecode reuses and extends cache to Live serving memory keeps changing
Request and weight flow
Control flow