Throughput Vs Latency
The serving tradeoff between aggregate completed work over time and the wait experienced by an individual request under concurrency.
What It Is
Throughput vs latency names the tension between two serving views that often move in opposite directions under load. Throughput is aggregate completed work over time. It counts how much model output the system finishes across all active requests in a chosen window—tokens per second, requests completed per minute, or similar fleet-level rates. Latency is the wait one individual request experiences from send through visible progress: time in queue before work starts, pauses between streamed tokens, and total wall clock until the answer feels done. The tradeoff appears when raising utilization improves throughput while stretching waits for some users.Why It Matters
Operators and readers often see both numbers move in opposite directions under load. A dashboard can show rising tokens per second while chat replies feel sluggish. Understanding the tradeoff separates healthy utilization from overload and clarifies why tuning for one metric can harm the other.What Changes Under Concurrency
Higher concurrency and larger batches are the most common reason the two views diverge. More simultaneous requests give the scheduler more work to pack onto each accelerator step, which raises aggregate completed work. Each request may wait longer before its turn starts or between streamed tokens because shared devices, memory paths, and batch formation time are finite. A deployment is not failing when throughput climbs while one reader waits longer—that pattern can mean the system is using capacity efficiently. The question is whether individual waits still meet the experience targets operators and product teams care about.Runtime Pressure And Cost
Queueing is the wait before model work begins. A request may sit in an admission queue while earlier jobs finish prefill, while memory headroom is checked, or while the scheduler decides which work is safe to start. That idle time does not count toward generated tokens, but it is part of the latency one reader feels. Under load, deeper queues can let the fleet finish more total work per second even as each newcomer waits longer at the door. Batching windows add a deliberate pause so compatible requests can run together. The scheduler may hold a request for a few milliseconds to gather enough work for one accelerator step. That short formation wait raises aggregate throughput because fixed per-step overhead is spread across more tokens or requests. The same pause stretches time to first token or the gap before the next decode turn for requests that arrived just before the window closed. Scheduling contention appears when many active requests compete for the same accelerators, memory pools, or worker threads. A decode turn deferred for a larger batch, a prefill burst that blocks new admissions, or a fairness policy that rotates long streams can all improve fleet utilization while making one stream wait its turn longer. Throughput rises when shared devices stay busy; individual latency rises when your request is not the one chosen next. Memory bandwidth pressure and live serving state set practical ceilings on both views. Each decode step reads growing key-value cache state, and concurrent sequences multiply that traffic. When bandwidth or resident cache size saturates, token cadence slows and aggregate output can flatten even though compute still looks available on paper. Live state—how many sequences are active, how long contexts have grown, and whether memory is near its limit—changes the tradeoff from minute to minute. Cost per completed unit is the economic mirror of the same tension. Higher utilization spreads fixed hardware cost across more finished tokens or requests, which lowers cost per token when throughput climbs. Pushing utilization further often means longer queues, wider batch windows, or accepting choppier streaming for some readers. Operators balance that savings against latency targets: a deployment that maximizes completed work per dollar may still miss product goals if individual waits exceed what users tolerate.Simple Example
Imagine ten chat requests arriving close together. The scheduler groups them into one larger batch so the accelerator stays busy and total tokens per second climbs. Each user may wait longer in queue and between streamed tokens even though the system completes more aggregate work during the same wall-clock window. If the same server handled one request at a time, that lone reader might see snappier gaps between tokens. Aggregate throughput would fall because the device idles between requests and never amortizes fixed per-step cost across concurrent streams.Common Confusions
Throughput vs latency is not a single latency metric like time to first token or inter-token latency, and it is not the same as tokens per second alone. Tokens per second measures aggregate output rate. Individual latency metrics measure one request's waits. The tradeoff explains how those views can diverge under concurrency, batching, and shared serving resources. High throughput does not guarantee a fast experience for every reader, and low latency for one request does not prove the fleet is using its capacity well. Both views answer different questions and both matter when judging serving behavior.Go Deeper Into Serving
The sections above explain why aggregate throughput and individual latency can move in opposite directions. Follow the links below to the serving pages that define each metric, execution phase, and policy that shapes the tradeoff under load.References
- Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, https://www.usenix.org/conference/osdi22/presentation/yu.
- Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.