AI model offloading lets a system run a model whose complete working set does not fit in fast accelerator memory. The runtime keeps performance-critical components on the accelerator and places other weights, cache or layers in a larger but slower tier.
Offloading trades capacity for transfer overhead. It works best when the moved component is accessed infrequently or can be transferred efficiently, while repeated movement across a slow bus can reduce inference speed substantially.
ELI5
Model offloading moves some AI model weights or working data out of scarce accelerator memory and into larger but slower system memory or storage. This can make a model run even when its complete working set does not fit on the accelerator.
For example, a local runtime can keep frequently used layers on a graphics processor while storing a large embedding table in ordinary memory. The model fits, but moving data between memory tiers can slow each response.
Where can AI model components be offloaded?
They can move from accelerator memory to ordinary system memory, or from memory to storage in more aggressive capacity-saving configurations.
Does model offloading make local inference slower?
It often adds latency because slower memory and transfers are involved, although careful placement can make a large model runnable with acceptable performance.
