Memory bandwidth measures how much data can move between memory and a processor in a given time. During artificial intelligence inference, model weights and intermediate values must be supplied to compute units quickly. If those units wait for data, adding more arithmetic capacity will not produce a proportional speed increase.
Large language models can be especially bandwidth-sensitive during token-by-token generation because substantial model data may need to be accessed for relatively little computation. Quantization, caching, smaller active parameter sets, and hardware with faster or unified memory can reduce this bottleneck. Capacity and bandwidth are different: a machine can hold a model in memory yet still run it slowly if transfer rates are limited.
