A graphics processing unit communication bottleneck appears when devices spend substantial time transferring data or waiting for other devices. The limiting link may be local device memory, an in-machine interconnect, a peripheral bus, or the network between cluster nodes.
Mitigation includes reducing communicated data, changing the partition, overlapping transfer with computation, using faster collectives, and matching operations to the correct link. Optimization must measure the actual hierarchy because different transfer paths have very different latency and bandwidth.
ELI5
A graphics processing unit communication bottleneck happens when GPUs spend too much time moving data or waiting for each other. The connection becomes the speed limit even though the processors could calculate faster.
For example, several GPUs may finish their local work quickly but wait while updates cross a slower network link. Engineers can reduce transferred data, change the partition or overlap communication with computation after measuring the actual bottleneck.
