Why Building an Eval Platform Is Harder Than It Looks - Braintrust

AI Engineer17m 47s
Hossein Niazmandi beside the blue and white headline “AI EVALUATIONS - BEYOND SPREADSHEETS” on a black background.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Hossein Niazmandi distinguishes evaluations before production from observability after deploymentAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it.. He explains why nondeterministic model behavior makes both necessary, then decomposes a basic evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. into test inputs, execution outputs and scores. His example illustrates how curated testsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. can help compare versions without proving an agent will perform reliably on real traffic.

    Hossein Niazmandi traces a progression from spreadsheets through custom interfaces and prompt experiments to production traces. The useful feedback loopAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior. turns observed failures into reproducible test cases, measures proposed changes and watches deployed behavior for new regressions. Engineers, product managers and domain specialists need different interfaces to participate in that process.

    Hossein Niazmandi describes the engineering burden behind the full systemAn AI evaluation platform provides shared infrastructure for defining, running, comparing and tracking AI evaluations.: large nested traces, full-text search, aggregate analytics, asynchronous scoring, annotations and access controls. He considers agents as additional users of evaluation tools while retaining human judgment over improvements. Braintrust supplies his examples, but the talk's central systems challenges apply beyond any single vendor.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools100
    • Models & learning25
    • Infrastructure & compute100
    • Research & evaluation100
    • Safety & security25
    • Business & productivity75
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Hossein Niazmandi beside the blue and white headline “AI EVALUATIONS - BEYOND SPREADSHEETS” on a black background. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 6 October 2026 and duration 17m 47s.

    Hossein Niazmandi argues that reliable agent evaluation requires a continuous production-feedback loop and substantial data infrastructure, not merely a spreadsheet interface.