Annabell Schäfer and Lotte Verheyden distinguish healthy infrastructure from correct agent behavior. Their engineering loop combines online tracing and monitoring with offline datasets, experiments and evaluation. They emphasize complete observations of model calls and tools, alongside latency, cost, prompt versions and the context needed to explain a failure.
Annabell Schäfer demonstrates monitoring with a small IT-support application. She maps conversation history and the latest user message into an LLM-based disagreement evaluator, adds a deterministic all-caps check as a possible frustration signal, and compares requests with the system prompt to flag work outside the application’s scope. These are task-specific signals, not definitive measures of correctness or user emotion.
Annabell Schäfer seeds example traces and uses a coding agent with Langfuse access to investigate recurring problems. The presentation reports retrieval failures, additional tool retries, latency and evaluation-coverage gaps. A prepared report and a newly returned report are shown separately; their findings illustrate an investigation workflow rather than an independently verified production audit.
Annabell Schäfer and Lotte Verheyden argue that the loop requires continued human review, maintained evaluation criteria and representative datasets. Interesting failures should become offline tests before changes are deployed. The recorded workshop concentrates on tracing and monitoring, so it does not establish that the proposed code change was successfully validated or that the complete offline deployment gate was implemented.
Watch on YouTube
