Latency measures the time from a request or completed input to the usable output. Interactive speech tools need short delays so recording, transcription and text insertion feel like one continuous action.
Performance depends on model size, hardware, runtime, input length and application overhead. A fair comparison uses the same device and representative inputs rather than treating one demonstration as a universal speed result.
ELI5
Low-latency AI inference means getting an answer from a model with very little waiting. The model may be powerful, but it also needs suitable hardware, a nearby server, and an efficient setup to respond quickly.
For example, a coding assistant that suggests the next line almost immediately feels interactive. If every suggestion takes several seconds, the same assistant interrupts the programmer's flow even when its answers are good.
What affects AI inference latency?
Model size, hardware, runtime, input length, memory transfers and application overhead all affect it.
Why is low latency important for speech transcription?
A short delay makes speaking and receiving editable text feel like one responsive interaction.
