Nan Jiang identifies a mismatch in reinforcement-learning post-training: trainers benefit from one tightly coupled cluster with fast interconnects, while rollout generation behaves more like a fleet of serving jobs and could use scattered GPU capacity. Keeping both inside one cluster forces the elastic rollout workload to inherit the trainer's hardest capacity constraint.
Nan Jiang proposes leaving backpropagation and collective communication in the trainer cluster while moving coherent rollout-serving islands across regions and providers. The interface between them is deliberately small: the trainer publishes an immutable policy version, and each rollout island returns trajectories, version information, log probabilities, and other replay metadata.
Nan Jiang argues that full checkpoints are the wrong synchronization unit. Because small optimizer steps often do not cross the rounding boundaries of BF16 or lower-precision served weights, consecutive rollout-visible versions can remain bit-identical for most parameters. A lossless patch can therefore transmit changed positions and replacement bits instead of hundreds of gigabytes of unchanged weights.
Nan Jiang presents Modal's Stitch implementation as an asynchronous, framework-agnostic versioning layer for globally autoscaled rollout engines. The talk reports substantial transfer reduction and internal sparsity measurements, while leaving open whether the same behavior holds with Muon, how far fully asynchronous reinforcement learning scales, and whether the approach generalizes beyond post-training.
Watch on YouTube
