On-device inference is the runtime stage in which a trained model processes input on the user's device. A mobile application might turn speech into text, classify an image, or generate a suggestion using the phone's CPU, GPU, neural processor, or another local accelerator. The model is already trained, so the device is applying learned parameters rather than performing the full training process.
Local inference can make an application responsive, available offline, and less dependent on accounts or cloud capacity. Its design must account for model download size, memory, battery use, thermal limits, device compatibility, and update delivery. Quantization, pruning, distillation, and hardware-specific runtimes can make models smaller and faster, but these optimizations may affect accuracy and need task-specific evaluation.
Acronyms and aliases
local inference synonymedge inference variant
General terms
Related terms
Frequently asked questions
How is on-device inference different from model training?
Training learns model parameters from data and usually requires substantial compute. Inference applies the trained parameters to new input to produce an output.
Does on-device inference require an internet connection?
Not for the model computation itself when the model and runtime are already installed, although the application may still use the internet for syncing, updates, or other features.
Why compress a model for on-device inference?
Compression can reduce download size, memory use, latency, and energy consumption so the model works within the constraints of phones and other edge devices.
