Benedikt Sanftl and Burak Cemil Özafşar describe an offline loop of building and evaluating agents and an online loop that monitors production tracesAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it. and feeds failures back into improvementsAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior..
They discuss platform-independent specifications, evaluation datasetsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria., actionable criteria and calibrating language-model judges. They argue that evaluation suites develop over time as teams encounter real failures and edge cases.
The speakers demonstrate Mutagent’s research-preview diagnostic agent, which samples traces, groups failure modes, explains possible causes and offers remedies for human reviewHuman-in-the-loop AI keeps a person involved in reviewing, correcting, approving, or guiding an AI system's work.. They describe a broader optimization loop that tests changes against an evaluation suite.
Watch on YouTube
