Aparna Dhinakaran explains why a fixed grading rubric can miss failures in a multi-step agent workflow. Tool useAI tool calling lets a model request a structured operation from software instead of only returning natural-language text., loops, changing interfaces and intermediate decisions all affect the outcome, so evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. needs to examine the trajectoryAn agent trajectory is the sequence of actions and observations an agent produces while working on a task. and its context rather than score only a final answer.
She proposes agent-based judges as a complement to deterministic checksDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state. and LLM-based scoring, not a replacement for either. Production tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. can reveal recurring tool inefficiencies and other behavioral patterns, giving engineers evidence for proposed fixes. The talk presents a development direction rather than claiming that adaptive judges eliminate evaluation risk.
Watch on YouTube




