Annabell Schäfer explains why green system metrics can coexist with an agent that falsely claims to have issued a refundAgent completion verification checks observable evidence that an agent achieved the requested result instead of accepting its claim that the work is finished.. The workshop separates tracingAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it., monitoring, representative datasets, controlled experiments and evaluationEvaluation measures how well an AI system performs against defined tasks, criteria and failure conditions using repeatable evidence., with domain experts helping define expected behavior rather than relying only on plausible final answers.
Doneyli De Jesus demonstrates a fictional banking assistant with masked personal information, tool traces and explicit guardrail scores. The examples distinguish ordinary information requests from cases that should be escalated or refused. The data shown is fictional, not customer banking data, and the demonstration is not a guarantee of regulatory compliance.
The practical workflow turns curated production examples and expert annotations into evaluation datasets. Doneyli De Jesus describes identifying failure categories from reviewer comments, combining deterministic checks with model-based judges and running repeatable experiments after prompt or configuration changes. The example gate reports thresholds and retains evidence; it does not automatically merge a passing change into production.
Annabell Schäfer closes by distinguishing useful automation from a vague request to fix an agent without supervision. Expert judgment remains important for ground truth and acceptable limits, while instrumentation and repeatable experiments can be automated. The final questions discuss self-hosting and calibrating model judges against representative human labels over time.
Watch on YouTube
