Fuad Ali explains how tracing helps teams inspect tool calls, model responses and failures in nondeterministic agents. He demonstrates a shopping assistant with a deliberately introduced search bug and shows how a monitoring agent can group failures and propose a code change.
Fuad Ali describes adding failures to a growing regression dataset, designing evaluations and replaying cases against a proposed fix. He compares deterministic checks with model-based evaluations and discusses the costs and sampling choices involved in monitoring live traces.
Fuad Ali emphasizes human judgment when confirming issues, curating test cases, reviewing changes and deciding whether to deploy. He also discusses experimental environments, external dependencies and the limits of using proxy metrics to judge whether a fix helps users.
Watch on YouTube
