Tim Scarfe joins Benjamin Crouzier, Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tešnar to examine ARC-AGI-3, an interactive benchmark in which an agent must infer the rules and goal of an unfamiliar game from raw frames and a limited action budget. Humans bring strong priors about players, mazes, keys and exits, while a model can identify many of the same concepts yet still act inefficiently or settle on the wrong objective.
The team describes a harness that combines language-model reasoningAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. with multiple representations of the game state, code execution, structured tools and reward shaping. Their experiments show that models benefit from human concepts encoded in language and color, but long-context reasoning remains brittle: once an agent forms a bad hypothesis, it can keep interpreting every later observation through that frame instead of revising its plan.
The discussion separates competence from performance and asks what a high benchmark score would actually proveEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. Action efficiency limits brute force, while unseen games require general strategies rather than memorized solutions. The team expects progress to combine detailed harness engineering, reinforcement learning and stronger base models, with the most valuable result being reusable ways to acquire abstractions rather than a system tuned to a fixed set of games.
Watch on YouTube
