Multi-graphics processing unit computing can distribute model weights and computation across several accelerators or place separate workloads on different devices. It expands total capacity and can provide an upgrade path for a workstation.
Performance does not scale automatically with device count. The runtime must support the model and hardware arrangement, while communication links between graphics processing units can become a bottleneck when data moves frequently.
ELI5
Multi-graphics processing unit computing means using several graphics processors for one large job or for several jobs at once. The work can be divided so a model that is too large for one device can use the combined capacity of several devices.
For example, two graphics processors might each hold part of a large model. They must exchange information as the model runs, so slow connections between them can reduce the speed gained from adding the second device.
Can memory from several graphics processing units be combined?
Software can partition a model across devices and use their combined capacity, but each device still has separate memory and transfers add overhead.
Does adding a second graphics processing unit double inference speed?
Usually not. Scaling depends on model partitioning, interconnect speed, workload shape and whether the serving software uses both devices efficiently.
