Artificial intelligence inference latency measures how long a model takes to respond during use. Teams may track time to first token, time per generated token or total completion time depending on the application.
Latency depends on model size, quantization, active parameters, context length, hardware, memory bandwidth and software. A smaller quantization can be faster, but implementation and hardware support determine the actual result.
