Researcher Aden Barton describes a Mercor accounting study comparing models and human participants on simplified benchmark tasksEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. He reports the study’s findings and distinguishes performance on a task from the full range of work in a profession.
The interview examines benchmark coverage, limitations in representing real work and the difference between automation and human-AI collaboration. The study’s results and productivity implications are attributed to the researcher rather than generalized to all accountants.
Watch on YouTube
