Phil Hetzel of Braintrust contrasts traditional service observability, such as latency and errors, with the additional challenge of assessing the quality of non-deterministic agent behaviourAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it..
He discusses inspecting rich traces, using human judgement and evaluations, and finding patterns in large collections of agent interactions. He describes how different teams need different views of the same system.
Hetzel presents clustering and annotation as ways to understand failures and improve evaluation, while emphasising the practical challenges of monitoring complex agent systems.
Watch on YouTube




