Parth Asawa defines continual learning as sample-efficient online learning that remains stable over long horizons. Standard language-model benchmarks reset state between independent tasks, so they measure point capabilities without revealing whether a system improves through experience.
A useful continual-learning benchmark needs headroom for online adaptation, shared latent structure across task instances and a realistic learning signal such as reward, error messages or textual feedback. The proposed evaluation measures reward, learning gain and cost rather than collapsing performance into one number.
Learning gain compares a stateful run with a baseline that resets the system between every task instance. This isolates improvement caused by earlier experience from the starting strength of the underlying model, while reward and cost preserve the other practical dimensions of system quality.
The benchmark spans six domains, including database exploration, software adaptation, epidemiology, signal processing, strategic game play and sales prediction. Database migration supplies concept drift: a capable learner must retain useful schema knowledge, discard stale assumptions and adapt efficiently to changed columns and formats.
Initial results show that simple in-context learning can outperform more expensive context-management systems on medium-horizon tasks. The observed failures cluster around stability, where prior feedback is forgotten, and plasticity, where a system rejects relevant new evidence; the longer-term research direction is to design model architectures and training processes for continual learning from the outset.
Watch on YouTube
