Collective communication provides standard operations for moving or combining data across many participants. AI workloads use collectives to distribute parameters, aggregate gradients, exchange activations, and keep parallel computation synchronized.
Ordering matters because all participants must enter compatible operations in a consistent sequence. Mismatches can deadlock or produce incorrect results. Performance depends on message size, topology, algorithms, overlap, and whether the selected collective matches the workload's partitioning.
ELI5
Collective communication is a standard way for a group of processors to exchange or combine data together. AI training uses these operations to share model information and keep parallel work synchronized.
For example, an all-reduce operation can combine gradient updates from many accelerators and give the result back to every participant. All processors must enter compatible operations in the same order, or the job can wait forever or produce incorrect results.
What are common collective communication operations?
Common operations include broadcast, reduce, all-reduce, gather, all-gather, scatter, reduce-scatter, and all-to-all.
Why does collective ordering matter?
Participants must call compatible collectives in a matching sequence or they may wait indefinitely, exchange the wrong data, or corrupt results.
