Time To First Token
The serving latency metric that measures how long a reader waits from request start until the first generated token appears.
What It Is
Time To First Token (TTFT) is the elapsed time from when a serving system accepts a generation request until the first newly generated token is ready to stream back. It includes request admission or queueing, prompt prefill, KV cache setup or reuse, and the first decode step that produces the opening token.Why It Matters
Readers feel TTFT as the awkward silence before a chat reply begins. Long prompts, busy queues, cold starts, and cache misses all stretch that pause even when later tokens arrive quickly. Operators track TTFT separately from inter-token latency and throughput because it answers a different question: how long until streaming starts.Simple Example
Imagine sending a long document and waiting for the model to start answering. The timer for TTFT starts when the request enters the server queue and stops when the first answer token is emitted. Everything before that first token—reading the prompt, building cache entries, and running the first generation step—counts toward TTFT.Common Confusions
TTFT is not the same as inter-token latency, total response time, or tokens per second. Inter-token latency measures the gap between tokens after streaming has begun. Total response time includes the full answer. Throughput counts how many tokens or requests the system handles over time. A model can have low TTFT but slow streaming, or the opposite.Serving Path
TTFT spans the gap between sending a prompt and seeing the first streamed token. Follow prefill for the prompt-processing burst, decode for the repeated next-token loop after streaming starts, and the prefill/decode split for how serving systems stage the two phases.References
- Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, https://www.usenix.org/conference/osdi22/presentation/yu.
- Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.