Ronak Malde argues that continual learning should use the enormous stream of production inference traces rather than depend on increasingly expensive benchmark environments. Current approaches suffer from mismatched task distributions, off-policy sampling, costly parallel rollouts and sparse sequence-level rewards.
The proposed on-policy self-distillation method turns the student model into its own stronger teacher by adding privileged guidance to the teacher prompt. Training then matches the unguided student's token probabilities to the guided teacher, preserving on-policy data while extracting dense feedback from a single rollout instead of a large rollout group.
Short tasks benefit from better token efficiency and broader shifts in the output distribution, but long tool-using trajectories expose serious failures. When student and teacher paths diverge, the teacher repeatedly favors hedging words and pulls the model into a poor local compromise; step-level divergence weighting limits training to portions of the trajectory where correction remains meaningful.
Privileged hints create a second risk by leaking answers that would not be available during real deployment. Filtering can convert direct solutions into realistic process guidance, while residual guidance compares partial and complete hints to avoid shifting the student toward behavior far outside its original distribution.
Together, these controls are presented as a scalable step toward systems that learn from online tasks with one rollout and per-token feedback. The closing vision is still provisional: present systems rely partly on offline batch updates, and genuinely continuous co-adaptation between models and their agent harnesses remains an open research and infrastructure problem.
Watch on YouTube
