Inter-Token Latency
The serving latency metric that measures the delay between streamed output tokens after generation has started.
What It Is
Inter-token latency is the elapsed time between consecutive streamed output tokens after generation has started. The first gap still belongs to time to first token (TTFT); inter-token latency begins once the opening token is visible and covers every later pause until the reply ends. Each interval is one decode step in practice—the model reads the growing key-value (KV) cache, runs the next-token pass, and hands the result to the stream.Why It Matters
Readers feel inter-token latency as the rhythm of a streaming reply after the first token appears. A short TTFT followed by sluggish gaps still feels like a slow answer, and a snappy cadence can make a long reply feel responsive even when the first token took longer. Operators track inter-token latency separately from TTFT and aggregate throughput because it answers a different question: how steady is token delivery once streaming has begun.Simple Example
Imagine a chat answer that begins streaming and then prints twenty more tokens. TTFT ends when the first answer token appears. Inter-token latency is the pause between token 1 and token 2, between token 2 and token 3, and so on through the rest of the stream. Summing those gaps describes how the reply felt; it is not the same as dividing total tokens by total wall time.Common Confusions
Inter-token latency is not time to first token, total response time, or tokens per second. Time to first token measures the wait before streaming starts. Total response time covers the full answer from request to finish. Tokens per second is an aggregate throughput figure—often averaged across many tokens, requests, or a benchmark window—and can look healthy even when individual gaps feel uneven. A system can have low TTFT but choppy streaming, fast per-token gaps but low overall throughput under heavy load, or the opposite.How Serving Latency Metrics Compare
Time to first token measures elapsed time from request start until the first streamed output token appears. Readers feel it as the awkward pause before the answer begins printing. Prefill cost, prompt length, queue wait, cold start, and cache misses before decode begins usually move it. Inter-token latency measures elapsed time between consecutive streamed output tokens after generation has started. Readers feel it as the rhythm or choppiness of the answer once streaming is underway. Decode work per step, KV cache reads, memory bandwidth, scheduler contention, and batching decisions usually move it. Tokens per second measures aggregate output rate—tokens completed over a window, often averaged across many tokens or requests. Readers feel it as how fast the system seems overall, even when individual gaps feel uneven. Batch size, device utilization, queue depth, request mix, and how much work shares each accelerator step usually move it.What Affects Token Cadence
Each inter-token interval is shaped by the decode work for one new token, the KV cache reads that step requires, and how quickly memory bandwidth can supply that state. Scheduler contention matters when many active requests compete for the same accelerators. Batching decisions—fixed batch sizes, continuous batching, or dynamic regrouping—change how often a request waits its turn. The active request mix also shifts cadence: prompt-heavy bursts, long decode chains, and cache-heavy conversations do not share the same per-token cost.Go Deeper Into Serving
The sections above name the factors that move token cadence. Follow the links below to the serving pages that explain each stage, system choice, or deployment tradeoff in more detail.References
- Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, https://www.usenix.org/conference/osdi22/presentation/yu.
- Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.