Batching
A serving system that groups multiple requests or decode steps together so accelerators stay busier, raising throughput while often adding wait time for each individual request.
Opening summary
Batching is the serving habit of waiting just long enough to run several requests or token-generation steps together instead of one by one. That denser work usually improves hardware utilization and tokens per second, but it can also make one reader wait longer before service begins.
At a glance
Released
July 2022
System type
Serving
Related models
No related models listed yet.
Related modules
No related modules listed yet.
What It Is
This is a serving system, not a model architecture change. It decides when separate requests or decode steps should be combined into one larger unit of work so the runtime can use the accelerator more efficiently.Where It Sits
Batching sits around the execution path for prefill and decode. The model still computes the same attention and feed-forward layers, but the serving layer decides whether those layers run for one request at a time or for a packed group that shares the same kernel launches, memory movement, and scheduling window.How It Works
Incoming work enters a queue, the runtime collects compatible items into a batch, and the accelerator runs that grouped work in one denser pass. During prefill the grouped work may be several prompts at once. During decode it may be one next-token step from many active sequences. After that shared step finishes, each request continues on its own output path even though it just shared compute with its neighbors.System flow: how a batch forms and runs
Legend: the top and bottom boxes are per-request handling steps, while the middle boxes are the shared batch-formation and shared execution stages.
Batching System Flow
Queue requestsForm a batchRun shared model stepReturn per-request outputsQueue requests to Form a batchForm a batch to Run shared model stepRun shared model step to Return per-request outputs
Request and weight flow