Laurie Voss uses a deliberately imperfect financial-analysis agent to show why an agent that appears useful can still fail unexpectedly. TracesAI agent observability makes an agent's state, actions, tool use, failures, resource use, and outcomes visible enough to understand and operate it. expose its model calls, tool choices, inputs and outputs, while evaluationsEvaluation measures how well an AI system performs against defined tasks, criteria and failure conditions using repeatable evidence. judge particular aspects of the result. He begins with reading failures and grouping their causes before deciding what to measure, distinguishing retrieval problems, unsupported claims, reasoning errors and ambiguous instructions.
Laurie Voss combines inexpensive deterministic checks with LLM judgesA large language model as a judge is an evaluation method in which a language model scores, compares or critiques another system's output using stated criteria. and human review. A ticker-name check catches simple scope errors, while a meaning-based judge needs the right question and evidence. A correctness judge that relies on outdated model knowledge wrongly rejects fresh information; a faithfulness judge becomes more useful when given the original research and search results instead of only the agent's notes.
Laurie Voss makes the evaluator itself an object of testing. Explicit criteria, clearly separated instructions and data, and human-labeled examples help define the task, but examples can also encourage overfitting. He separates development labels from held-out tests, discusses false alarms and missed failures, and shows how a stricter rubric can disagree with inconsistent human labels. A small demonstration dataset is not proof of production reliability.
Laurie Voss compares changes on the same tasks and evaluation criteria, using repeated runs to expose nondeterminism. A prompt change greatly improves whether the toy reports give a clear recommendation but does not fix their poor faithfulness. A model upgrade changes consistency and cost while introducing an overly conservative response pattern, illustrating why one improved score can conceal a different weakness.
Laurie Voss turns reliable capability checks into regression evaluations and demonstrates release gates in a test runner and CI. He distinguishes mandatory guardrails from quality targets and cautions against running an expensive whole-agent suite on every commit. The workshop closes with selective production sampling and trend monitoring as the next step; its example analyst is a teaching device, not a trustworthy investment adviser.
Watch on YouTube
