Epoch AI’s Greg Burnham discusses benchmark trendsA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. with Sam Charrington, including how an aggregate index combines results across tests and why progress may reflect both targeted training and generalization.
Burnham describes the shift from math exams to open problems and discusses reported AI-assisted mathematical resultsAI-assisted scientific discovery uses AI to support hypothesis generation, experiment design, analysis, simulation, literature work, and interpretation while researchers retain responsibility., their reliance on prior human work, and the difficulty of judging whether a solution contains a genuinely new ideaHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences..
They also explore tests for research abilityEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria., learning from repeated play, workplace tasks and visual reasoning. Burnham frames the results as current observations rather than guarantees about future systems.
Watch on YouTube
