Gurvir Singh and Rayan Garg distinguish human time horizons from model measures such as tokens, steps and tool calls. Human completion time depends on expertise and methodology, while model measures vary with the model and harness. They argue that these views should be considered together because neither alone establishes how difficult a task is for an agent.
The talk separates tool coordination, changing environment state and ambiguity as dimensions of task complexity. Simply chaining unrelated tasks makes a trajectory longer without necessarily testing deeper capability. More meaningful tasks require early decisions to influence later work, and leave enough uncertainty for agents to explore rather than follow one prescribed solution.
Gurvir Singh and Rayan Garg describe judges as agents that may need to inspect both the final environment and the path takenAgent completion verification checks evidence that an agent achieved the requested outcome rather than merely producing intermediate work. to reach it. A deployment-repair example shows why a judge must check logs and outcomes, not just accept tool-call claims. The proposed safeguards include read-only access for judgesLeast privilege gives a person or automated system only the access required for its current authorised work., scrutiny of reward hacking, and queryable records of long trajectoriesAn execution trace records the sequence of operations and results produced while a system carries out a task..
The closing discussion connects rubric design to learnability and evaluation qualityEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. It favors combining deterministic checksDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state. with model judges, allowing multiple valid solutions, testing agreement with experts, and examining the breadth of task coverage. The presenters criticize narrow or saturated finance benchmarks and use company data as an illustration; those comparisons are presented as their own assessment rather than independently verified results.
Watch on YouTube




