Memory

A serving system concern that covers the live state the runtime must keep or move while answering requests, including model weights, KV cache growth, allocator overhead, and memory bandwidth limits.

Opening summary

Serving memory is the live state an inference runtime must keep nearby or move quickly enough to answer requests. It is not just the advertised capacity number on a device spec sheet. What matters in practice is which weights stay resident, how the key-value cache grows as sessions stay active, and how fast the system can move that state without stalling the next token.

At a glance

Released

July 2022

System type

Memory

Related models

No related models listed yet.

Related modules

No related modules listed yet.

What It Is

Memory is a serving-system concern because a model can only answer requests when the runtime can hold or fetch the state needed for the next step. That live state includes resident model weights, the growing key-value cache, temporary activations, and the bookkeeping overhead required to place them on usable hardware.

Where It Sits

It sits underneath visible serving choices such as batching, routing, deployment, and prefill versus decode scheduling. Those systems do not change the learned model architecture itself. They decide how work is placed so the right memory stays available, close enough, and fast enough for the inference engine to keep serving.

How It Works

Before useful work begins, the runtime needs the model weights resident in a fast enough memory tier to execute the forward pass. Prefill usually creates the sharpest one-time burst of cache growth because the system has to read the whole prompt and write the first key-value cache entries for every token in it. Decode then reuses those entries and extends them token by token, so the pressure shifts from a big prompt-ingest burst to a longer-lived accumulation problem while active requests remain alive. That is why memory stays a live serving concern after load: the stack keeps balancing resident weights, cache growth, and free space for the next step instead of treating memory as a one-time loading problem. Memory capacity answers how much state can exist at once, while memory bandwidth answers how quickly the runtime can read or move that state. A system can have enough total bytes and still miss targets when the next layer cannot pull weights or active cache blocks fast enough. Fragmentation adds a second limit because enough memory on paper does not always mean enough usable memory in time.
Memory System Flow
Request and weight flow
Control flow
Weights dominate the starting footprint, prefill creates the first cache burst, and decode keeps the live memory total moving as requests stay active.

Practical Impact

Serving can look compute-rich and still slow down when memory is the tighter constraint. When weights are expensive to read, first-token latency rises because the runtime spends longer pulling the prompt path through the model. When many sessions stay active, inter-token latency and total throughput can flatten because each decode turn has to move more live cache state before useful math resumes. The practical result is that memory limits often show up as queue growth, lower safe concurrency, or unstable tail latency before they show up as a simple out-of-memory crash. Operators respond by reshaping work around the memory they actually have. Larger or denser batches can improve throughput because more useful tokens share one step, but they also enlarge live cache state and can delay a single request in queue. Spilling colder cache to slower storage keeps longer sessions alive, but every reload adds latency when decode needs those tokens again. Splitting prefill and decode lets one stage specialize for prompt bursts and another for long-lived token loops, but it adds handoff cost and depends on routing that keeps the right cache near the next worker. Deployment and routing decisions often exist for the same reason: they place requests on hardware tiers and serving paths whose memory shape matches the expected weights, cache growth, and latency target, not only to save compute.
textactiveservingmemoryapproxtextresidentweights+textKVcachebytes+textallocatoroverhead+textmovementslack\\text{active serving memory} \\approx \\text{resident weights} + \\text{KV cache bytes} + \\text{allocator overhead} + \\text{movement slack}

Tags

References

  1. Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, https://www.usenix.org/conference/osdi22/presentation/yu.
  2. DeepSeek-AI. "DeepSeek-V4 Technical Report." 2026.