The footprint includes model weights, caches, activations, buffers and runtime overhead. It changes with numeric precision, context length, batch size and whether components use accelerator or ordinary system memory.
Telemetry should measure peak use during a realistic workload. A model that barely loads can still run out of memory later when a long conversation expands its cache.
ELI5
A model memory footprint is the amount of working space a computer needs to load and run a model. This includes the model itself plus temporary information it keeps while processing a request.
For example, a model might fit on a laptop after its weights are compressed, but a long document could still make it run out of memory. The computer needs room for both the model and the extra working data created during the task.
Why can a model memory footprint grow during chat?
The runtime may retain an expanding attention cache and other state as the conversation becomes longer.
How should model memory use be measured?
Measure peak use with realistic context, settings and output length on the target runtime.
