Laurie Voss explains why nondeterministic agents require more than ordinary code inspection. Nested tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. expose prompts, tool calls, outputs, latency and cost; evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. add scores and explanations so teams can find recurring failure patterns without manually reading every interaction.
Laurie Voss works through a toy-store shopping agent with observability instrumentation and trace-analysis skillsAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work.. A broken price filter produces empty results, and the workshop demonstrates diagnosing and checking a correction. The demo also encounters setup problems and uses an existing solution, so it is an illustration of the workflow rather than proof of autonomous debugging success.
Laurie Voss describes periodically examining production traces to propose issues, evaluations and code changes. Humans remain responsible for accepting or rejecting those suggestionsHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences., while regression evaluationsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked. protect previously working behavior and guardrails test domain boundaries. He also discusses deployment and compliance limitations of the demonstrated tooling.
Watch on YouTube




