Artificial intelligence inference autoscaling adds or removes serving resources as request volume, queue depth or latency changes. Systems may scale prefill and decode workers separately because those stages have different resource requirements.
Model workers can be slow and expensive to start, so autoscaling must anticipate demand and preserve useful caches where possible. Policies also need limits that control cost and prevent unstable cycles of rapid expansion and contraction.
Acronyms and aliases
AI inference autoscaling acronymmodel-serving autoscaling synonymartificial intelligence inference autoscaling variant
General terms
Frequently asked questions
Which signals can trigger artificial intelligence inference autoscaling?
Useful signals include queue depth, accelerator utilization, request rate, latency percentiles, cache pressure and stage-specific capacity.
Why is model-serving autoscaling harder than ordinary web autoscaling?
Model instances use scarce accelerators, load large weights, hold valuable cache state and may require coordinated multi-device placement before serving traffic.
