Inter-token latency measures the pace of generation after the first output token appears. Lower values make streamed text or speech feel smoother, while high or inconsistent values create pauses during the response.
The metric depends on model size, hardware, memory bandwidth, batching and concurrent load. Tail percentiles such as P99 reveal rare slow intervals that an average can conceal.
Acronyms and aliases
ITL acronymtime per output token synonym
Related terms
Frequently asked questions
How is inter-token latency different from time to first token?
Time to first token measures response startup, while inter-token latency measures the spacing between tokens after generation has begun.
Why measure P99 inter-token latency?
The 99th percentile exposes severe generation pauses affecting a small share of tokens even when the average performance appears acceptable.
