Inter-token latency measures the pace of generation after the first output token appears. Lower values make streamed text or speech feel smoother, while high or inconsistent values create pauses during the response.
The metric depends on model size, hardware, memory bandwidth, batching and concurrent load. Tail percentiles such as P99 reveal rare slow intervals that an average can conceal.
ELI5
Inter-token latency is the delay between one generated token and the next after an AI response has started. Lower delay makes streamed text or speech feel smooth.
For example, an assistant can produce its first word quickly but then pause noticeably between later words. Average delay matters, and high-percentile measurements reveal rare slow gaps that users still experience.
How is inter-token latency different from time to first token?
Time to first token measures response startup, while inter-token latency measures the spacing between tokens after generation has begun.
Why measure P99 inter-token latency?
The 99th percentile exposes severe generation pauses affecting a small share of tokens even when the average performance appears acceptable.
