Jack Morris starts from a gap between the broad public knowledge learned during pre-training and the depth found in private, personal or long-tail expertise. Because a model cannot learn new events, company history or individual preferences after training, the talk asks how extra compute can internalize a fixed body of context.
With data volume constrained and training a model from scratch impractical, compute becomes the main remaining scaling axis. Naive next-token training can memorize a document collection yet fail to generalize, while key-value compaction only helps when the source material already fits inside the context window.
On-policy distillation, synthetic continued pre-training and unsupervised reinforcement learning offer more promising forms of knowledge transfer. Each still depends on a defined training set, however, so performance eventually saturates instead of preserving the open-ended gains associated with large-scale pre-training.
The proposed direction is a recursive process in which a model improves, generates harder or better training material from the same domain and then improves again. The desired result is progressively deeper understanding of private context, such as a person's communications, goals and preferences, as more compute is invested.
Watch on YouTube
