Tokens Per Second
Tokens per second (tokens/s, tok/s, or TPS) is a throughput rate: how many model tokens a serving system generates per second over the measurement window you choose.
What It Is
Tokens per second (tokens/s, tok/s, or TPS) measures how many model tokens a serving system completes per second over a chosen window. A token is the model's text unit—the piece a tokenizer splits text into before the network runs. The same visible sentence can map to different token counts across models or tokenizers, so throughput always refers to model tokens, not words, characters, or bytes.Why It Matters
Operators use tokens per second to see how much generation work a deployment can sustain. Higher throughput can mean better accelerator use or more concurrent answers, but the headline number depends on batching, prompt length, and whether the window includes prefill, decode, or both. Readers may care indirectly when capacity limits queue depth or streaming pace, even though throughput is usually averaged across many tokens or requests.Simple Example
If a server emits 120 generated tokens in two seconds of steady streaming, that slice averages 60 tokens per second. The same deployment might report a different rate over a longer window that also includes queueing, prompt prefill, or idle gaps between requests. The window you choose changes the number even when each individual token took the same time to produce.Common Confusions
Tokens per second is not the same as total request latency, time to first token, or inter-token latency. Latency metrics describe waits and gaps for one stream. Throughput describes how many tokens move through the system per second, often averaged across many tokens, requests, or a benchmark window. A deployment can show high aggregate throughput while one reader still waits a long time before streaming starts.How Serving Metrics Compare
Total request latency measures wall time from send until the streamed answer finishes. A reader feels it as the full wait for the complete reply, including queueing, prompt work, every token gap, and any tail cleanup. It answers how long one answer took end to end, not how many tokens the fleet averaged per second. Time to first token measures elapsed time from request start until the first streamed output token appears. A reader feels it as the pause before anything starts printing. Prefill cost, prompt length, queue wait, cold start, and cache misses before decode begins usually move it. A fast headline throughput number can still leave a long blank screen if TTFT is high. Inter-token latency measures elapsed time between consecutive streamed output tokens after generation has started. A reader feels it as the rhythm or choppiness of the answer once streaming is underway. Decode work per step, KV cache reads, memory bandwidth, scheduler contention, and batching decisions usually move it. Steady per-token gaps can still add up to modest throughput when each step is expensive. Tokens per second measures aggregate output rate—generated tokens completed over a chosen window, often averaged across many tokens or requests. A reader may feel it only indirectly as how quickly capacity seems to recover under load, because throughput is usually a fleet or benchmark figure rather than one visible gap. Batch size, device utilization, queue depth, request mix, and how much work shares each accelerator step usually move it.What Moves Tokens Per Second
Batch size can raise aggregate throughput by letting one accelerator step finish more generated tokens at once. That efficiency gain does not guarantee a smoother experience for one reader: a larger batch may improve tok/s while lengthening queue wait before streaming starts or changing how evenly tokens arrive mid-stream. The prefill/decode mix changes throughput because prompt processing and repeated output-token generation stress different parts of the system. A window dominated by long prompts may report lower generation throughput even when decode itself is fast, while a decode-heavy burst can push tok/s up until memory movement or scheduler limits appear. KV cache reads and writes plus memory bandwidth often cap decode-heavy serving. Each new token reuses growing cache state, so bandwidth and cache traffic can become the bottleneck even when raw compute headroom remains. That is why throughput can flatten as sequences lengthen or concurrency rises. Active model size shifts tokens per second through compute cost per step, how much state must move each decode, and whether the deployed hardware can keep weights and cache resident without spilling or oversubscribing memory. A smaller model on the same devices may report higher tok/s; a larger one may need more replicas, narrower batches, or different precision just to stay within memory limits.Go Deeper Into Serving
The sections above name the factors that move throughput. Follow the links below to the serving pages that explain execution policy, batching tradeoffs, scheduler contention, and the mechanics behind each throughput limit.References
- Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, https://www.usenix.org/conference/osdi22/presentation/yu.
- Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.