Building an AI Observability and Evaluation Workflow

AI Engineer1h 51m
Doug Guthrie beside the headline Observe Your AI on a black background.
1 VIEW
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Doug Guthrie builds an observability workflow around a support agentAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. using the OpenAI Agents SDK and Braintrust. He starts with tracing inputs, outputs and tool callsAn execution trace records the sequence of operations and results produced while a system carries out a task., then distinguishes deterministic checks, LLM judges and human calibrationHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences. as sources of quality signals.

    Doug Guthrie configures online scoring and topic analysis to find patterns beyond known test cases. The workshop covers custom facets, preprocessing, sampling, conversation grouping and querying trace metadata, using imported support traces to investigate record-lookup failures and other recurring issues.

    Doug Guthrie demonstrates bringing selected production examples back into evaluation datasetsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and using a coding agentAn AI coding agent is a tool-using AI system that can inspect, modify, and validate software within a repository. with the Braintrust CLI to investigate failures, propose changes and compare evaluation results. He emphasizes a reviewable improvement cycle rather than treating collected traces as useful on their own.

    The questions explore deployment models, access restrictions, scoring cost, changing test sets and remote evaluations. Doug Guthrie shows how parameterized evaluations can expose an agent to a playground while keeping code execution on a separate server.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows80
    • Coding & tools65
    • Models & learning0
    • Infrastructure & compute20
    • Research & evaluation100
    • Safety & security25
    • Business & productivity25
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Doug Guthrie beside the headline Observe Your AI on a black background. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 6 October 2026 and duration 1h 51m.

    Doug Guthrie demonstrates how production traces, evaluators and topic clustering can feed a tested improvement cycle for AI agents.