Artificial intelligence model offloading lets a system run a model whose complete working set does not fit in fast accelerator memory. The runtime keeps performance-critical components on the accelerator and places other weights, cache or layers in a larger but slower tier.
Offloading trades capacity for transfer overhead. It works best when the moved component is accessed infrequently or can be transferred efficiently, while repeated movement across a slow bus can reduce inference speed substantially.
Acronyms and aliases
AI model offloading acronymartificial intelligence model offloading variantmodel component offloading variant
Related terms
Frequently asked questions
Where can artificial intelligence model components be offloaded?
They can move from accelerator memory to ordinary system memory, or from memory to storage in more aggressive capacity-saving configurations.
Does model offloading make local inference slower?
It often adds latency because slower memory and transfers are involved, although careful placement can make a large model runnable with acceptable performance.
