Alfonso Graziano introduces golden datasets and scorers for assessing agent behavior, including tool calls as well as answers. He describes how missing tools, instructions or context can cause evaluation failures.
He demonstrates an optimization loop that proposes changes, runs evaluations and keeps improvements or rolls back regressions. He reports results from both a simple example and an internal production agent, while noting that the simple baseline was easy to improve.
Graziano describes collecting user feedback and traces, clustering failure modes and asking subject experts to validate priorities. He argues that specifications, tests, observability and human oversight support safer iteration.
Watch on YouTube
