Nan Jiang identifies a mismatch in reinforcement-learningReinforcement learning trains an AI system to choose actions using feedback about the results of its earlier choices. post-training: trainers benefit from one tightly coupled cluster with fast interconnects, while rollout generation behaves more like a fleet of serving jobs and could use scattered GPU capacityAI compute infrastructure is the hardware, facilities, networks, power, cooling, storage, and software used to train and run AI models.. Keeping both inside one cluster forces the elastic rollout workload to inherit the trainer's hardest capacity constraint.
Nan Jiang proposes leaving backpropagation and collective communication in the trainer cluster while moving coherent rollout-serving islands across regions and providers. The interface between them is deliberately small: the trainer publishes an immutable policy version, and each rollout island returns trajectories, version information, log probabilities, and other replay metadata.
Nan Jiang argues that full checkpointsAn AI model checkpoint is a saved version of model parameters and related state from a particular point in training or development. are the wrong synchronization unit. Because small optimizer steps often do not cross the rounding boundaries of BF16 or lower-precision served weights, consecutive rollout-visible versions can remain bit-identical for most parameters. A lossless patchDelta encoding represents changes relative to a known base version instead of storing or transmitting the complete updated data each time. can therefore transmit changed positions and replacement bits instead of hundreds of gigabytes of unchanged weights.
Nan Jiang presents Modal's Stitch implementation as an asynchronous, framework-agnostic versioning layer for globally autoscaled rollout engines. The talk reports substantial transfer reduction and internal sparsity measurements, while leaving open whether the same behavior holds with Muon, how far fully asynchronous reinforcement learning scales, and whether the approach generalizes beyond post-training.
Watch on YouTube
