Janie Gu explains why AI benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. can miss real work, become expensive to maintain, and produce inconsistent results across evaluation infrastructure. She describes problems Kaggle found while reproducing 65 benchmarks and outlines proposed isolation, logging and independent audit measuresIndependent evaluation assesses a system through reviewers whose judgment is sufficiently separate from the organization developing or promoting it..
Suyeol Yun presents MandateBench, which evaluates an agent on a task from an investment mandate through a memo. He describes grading the evidence and reasoningEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. behind an investment recommendation, including how an analyst handles conflicting financial figures.
The speakers present early results from ten tasksAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. and plans for a larger benchmark. The reported results are preliminary and do not establish general investment performance.
Watch on YouTube
