Simran Arora explains why multi-GPU communication has become a central performance bottleneck as AI workloads scale. GPU memory, NVLink, PCIe and cluster networking create a hierarchy of transfer costs that efficient kernels must coordinate rather than treating communication as a single interchangeable operation.
Simran Arora presents ParallelKernelBench, a suite of 87 realistic multi-GPU problems. The strongest tested frontier model solved 28 problems without iterative feedback, with 22 solutions outperforming a PyTorch and NCCL baseline. More samples raised correctness, but speed improvements plateaued, showing that search alone does not resolve the deeper optimization problem.
The models often reuse familiar patterns found in existing code, yet struggle with collective ordering, data partitioning, inter-SM and intra-SM scheduling, and choosing the right transfer mechanism. A multi-turn agent improves results by compiling, testing and revising kernels, but it also plateaus, suggesting that better hardware-aware reasoning is still needed.
Watch on YouTube
