AI inference repeatedly moves weights, activations and cached state through the memory system. When that transfer rate is lower than the processor's demand, the processor waits for data and additional arithmetic capacity brings little benefit.
Architectures that replace computation with large lookup tables can become especially dependent on bandwidth. Real speed therefore depends on where data resides and how quickly it can be delivered, not only on parameter counts or operation counts.
ELI5
Memory bandwidth is how quickly data can move between memory and a processor. A system may have enough memory to hold a large AI model but still run it slowly if the processor has to wait for model values to arrive.
For example, generating each token may require reading a large part of the model's weights again. Wider or faster memory connections can deliver that data sooner, while access patterns and software determine how much of the advertised bandwidth is actually used.
How can memory bandwidth limit AI inference?
The processor cannot use weights or activations until they arrive, so slow data movement can leave its compute units idle.
Is memory capacity the same as memory bandwidth?
No. Capacity is how much data memory can hold, while bandwidth is how quickly that data can be transferred.
