Quantization converts weights, and sometimes activations, from higher-precision numbers to compact formats. Smaller representations can let capable models fit into local memory and reduce the amount of data moved during generation.
Lower precision can reduce output quality or require specialized kernels. The best format depends on the model, workload, runtime and available hardware rather than the bit count alone.
ELI5
Model quantization stores and calculates some learned model values with fewer bits. This is like using a shorter numerical scale: it can make the model smaller, reduce memory use, and sometimes make inference faster on suitable hardware.
For example, converting model weights from a high-precision format to an eight-bit or four-bit format may let the model fit on a smaller device. The approximation can reduce quality on some tasks, so the quantized version must be tested rather than assumed to behave exactly like the original.
Why use model quantization for local AI?
It reduces weight size and memory traffic, which can make larger models usable on local hardware.
Can model quantization reduce quality?
Yes. Lower numeric precision introduces approximation, and the effect varies across models and tasks.
