Jason Mancuso argues that fine-tuning can help teams control agent quality, latency and cost. He contrasts supervised fine-tuning on recorded tracesFine-tuning adapts a pretrained model by updating its parameters on examples chosen for a particular task or subject. with reinforcement learning, in which agents explore alternative trajectories and receive rewardsReinforcement learning trains an AI system to choose actions using feedback about the results of its earlier choices..
He describes a training loop that runs inference rollouts and model updates asynchronously, noting that stale model versions and differences between training and inference systems can destabilize learning. He outlines a recipe combining group relative policy optimization, masked importance sampling, score centering and sampling-mask replay; stability claims are attributed to Modal’s tests.
Mancuso describes compressing model updates so rollout inference can run across distributed GPU fleetsAI compute capacity is the available ability of hardware and supporting systems to perform AI training or inference work over time., including different hardware types. He reports lower rollout costs in a specific run and discusses the trade-offs of fragmented capacity.
Watch on YouTube
