Hossein Niazmandi distinguishes evaluations before production from observability after deploymentAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it.. He explains why nondeterministic model behavior makes both necessary, then decomposes a basic evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. into test inputs, execution outputs and scores. His example illustrates how curated testsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. can help compare versions without proving an agent will perform reliably on real traffic.
Hossein Niazmandi traces a progression from spreadsheets through custom interfaces and prompt experiments to production traces. The useful feedback loopAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior. turns observed failures into reproducible test cases, measures proposed changes and watches deployed behavior for new regressions. Engineers, product managers and domain specialists need different interfaces to participate in that process.
Hossein Niazmandi describes the engineering burden behind the full systemAn AI evaluation platform provides shared infrastructure for defining, running, comparing and tracking AI evaluations.: large nested traces, full-text search, aggregate analytics, asynchronous scoring, annotations and access controls. He considers agents as additional users of evaluation tools while retaining human judgment over improvements. Braintrust supplies his examples, but the talk's central systems challenges apply beyond any single vendor.
Watch on YouTube




