From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 - Laurie Voss, Arize AI

AI Engineer42m 17s
Laurie Voss against a black background beside the blue-and-white headline “FIX AGENTS WITH FEEDBACK”.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Laurie Voss explains why nondeterministic agents require more than ordinary code inspection. Nested tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. expose prompts, tool calls, outputs, latency and cost; evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. add scores and explanations so teams can find recurring failure patterns without manually reading every interaction.

    Laurie Voss works through a toy-store shopping agent with observability instrumentation and trace-analysis skillsAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work.. A broken price filter produces empty results, and the workshop demonstrates diagnosing and checking a correction. The demo also encounters setup problems and uses an existing solution, so it is an illustration of the workflow rather than proof of autonomous debugging success.

    Laurie Voss describes periodically examining production traces to propose issues, evaluations and code changes. Humans remain responsible for accepting or rejecting those suggestionsHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences., while regression evaluationsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked. protect previously working behavior and guardrails test domain boundaries. He also discusses deployment and compliance limitations of the demonstrated tooling.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows90
    • Coding & tools70
    • Models & learning15
    • Infrastructure & compute15
    • Research & evaluation100
    • Safety & security35
    • Business & productivity20
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Laurie Voss against a black background beside the blue-and-white headline “FIX AGENTS WITH FEEDBACK”. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 5 October 2026 and duration 42m 17s.

    Laurie Voss shows how traces, evaluations and human-approved changes can turn real agent failures into a repeatable improvement and regression-testing loop.