Dat Ngo describes observability as a way to examine agent activity through traces, spans, sessions and pathsAn execution trace records the sequence of operations and results produced while a system carries out a task.. He explains how analyzing different trajectories can reveal latency or ordering problems.
Dat Ngo compares human feedback, labeled datasets, model judges and deterministic checksHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences.. He distinguishes evaluations of individual components from evaluations across multiple steps or sessionsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and discusses the cost of measuring everything.
Dat Ngo describes using collected traces or datasets to compare changes to prompts, models and orchestration. He demonstrates the ideas with his company’s tools and discusses possible future automation as a company goal.
Watch on YouTube
