Edward Hu talks with Sophia Dew about Mercor’s research into AI model training and economically valuable tasks. He describes an accounting benchmarkA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. in which models performed well on a narrow set of retrieval-heavy tasks, while emphasizing that a task is not an entire job.
Edward Hu explains why benchmark design needs fair rubrics and diverse examples. Overly specific grading can penalize a correct answer, while a model that lists many possible answers can exploit a grader that rewards any mention of the right one.
The discussion turns to more realistic environments with files, messages and other tools. Edward Hu describes Mercor’s open model training work and proposed directions for evaluating AIEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. in organizational settings; future releases are presented as plans.
Watch on YouTube
