Time to first token includes queueing, routing and the prompt prefill stage before output generation begins. It is a key measure of perceived responsiveness for interactive applications because the user sees no model output during this interval.
Caching a repeated prefix can reduce time to first token by avoiding prompt recomputation. Capacity planning, prefill scaling, model placement and request scheduling also influence the delay.
ELI5
Time to first token is the wait between sending an AI request and receiving the first piece of generated output. It includes queueing, routing, and processing the prompt before generation starts.
For example, two models may generate tokens at the same rate, but one feels slower because output begins several seconds later. Prompt caching, capacity, model placement, and scheduling can reduce that initial delay.
What does TTFT stand for?
TTFT is the acronym for time to first token, the startup delay before the first generated token arrives.
Does a low time to first token guarantee fast generation?
No. Startup can be fast while later tokens arrive slowly, so inter-token latency and total response time must also be measured.
