Inter-token latency measures the pace of generation after the first output token appears. Lower values make streamed text or speech feel smoother, while high or inconsistent values create pauses during the response.
The metric depends on model size, hardware, memory bandwidth, batching and concurrent load. Tail percentiles such as P99 reveal rare slow intervals that an average can conceal.
ELI5
Inter-token latency is the delay between one generated token and the next after an AI response has started. Lower delay makes streamed text or speech feel smooth.
For example, an assistant can produce its first word quickly but then pause noticeably between later words. Average delay matters, and high-percentile measurements reveal rare slow gaps that users still experience.
