Artificial intelligence model serving places a trained model behind an application or network interface so it can receive inputs and return outputs. The serving layer loads weights, schedules computation, manages memory and exposes the model to products or developer tools.
Serving optimizations can reduce latency and cost through batching, caching, quantization, parallelism and hardware-specific kernels. Good serving also handles scaling, monitoring and failures. A capable model may still be uneconomic if its serving stack consumes too much computation for each completed task.
Acronyms and aliases
model serving synonymAI model serving variant
Related terms
Frequently asked questions
How is model serving different from model training?
Training adjusts a model's parameters from data. Serving loads the trained model and operates it for real requests, focusing on response time, reliability, capacity and cost.
What makes AI model serving efficient?
Efficiency can come from request batching, optimized kernels, caching, reduced precision, sparse computation and careful use of accelerators. The best mix depends on the model and workload.
Videos explaining artificial intelligence model serving
