Continuous Batching
A serving system that keeps an active decode set refilled as requests finish or become compatible, so accelerators spend less time waiting for a whole batch to turn over.
Opening summary
Continuous batching is a serving technique for decode-heavy workloads. Instead of waiting for one fixed request batch to finish before starting another, the runtime keeps admitting new compatible work as slots open, which can keep the active decode set denser and reduce GPU idle time.
At a glance
Released
July 2022
System type
Serving
Related models
No related models listed yet.
Related modules
No related modules listed yet.
What It Is
Continuous batching is a serving system, not a model architecture change. Its job is to keep a decode batch alive over many turns by refilling open slots with new compatible requests instead of treating every batch as a sealed unit that must finish all together.Where It Sits
It sits inside the inference-serving runtime alongside request queues, batching policy, routing, inference-engine execution, and key-value cache management. The underlying model still computes the same layers. Continuous batching changes how the serving stack schedules decode work around those layers so active slots stay occupied between request completions.How It Works
Static request batching usually waits for one chosen batch to turn over before the scheduler admits more work. Continuous batching keeps a queue nearby, runs a decode step for the active set, removes requests that finish, and immediately refills those open slots with queued requests that can join the next compatible turn. The important behavior is incremental refill: the runtime does not rebuild service only at one fixed batch boundary, so decode work can stay denser across many turns.Continuous Batching System Flow
Queue compatible requestsKeep active decode set aliveRun next shared decode turnRemove finished requestsRefill open slots for the next turnQueue compatible requests to Keep active decode set aliveKeep active decode set alive to Run next shared decode turnRun next shared decode turn to Remove finished requestsKeep active decode set alive to Refill open slots for the next turnRefill open slots for the next turn to Run next shared decode turn
Request and weight flow
Control flow