Sally-Ann DeLucia explains why informal impressions cannot reliably detect regressionsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked. in agents with many valid outputs and multiple tool calls. TracesAn AI execution trace is a structured record of the steps, tool calls, state changes, and outputs produced during an AI workflow. reveal the steps and data behind an answer, while evaluationsAgent evaluation tests whether an AI agent completes tasks correctly, consistently, and within its required boundaries. test a clearly defined quality. Different questions may need deterministic code checks, model judgesA large language model as a judge is an evaluation method in which a language model scores, compares or critiques another system's output using stated criteria., annotations or production metrics rather than one universal score.
Sally-Ann DeLucia uses a travel-planning agent to turn vague expectations into observable requirements, such as answering a follow-up, respecting trip dates and keeping within a budget. She recommends starting with one quality, writing specific examples of success and failure, and selecting the root or tool spans that actually contain the evidence the evaluator needs.
Sally-Ann DeLucia then builds a judge prompt, maps dataset fields to its inputs and inspects labels and explanations before iterating. Comparing those judgments with human annotationsData annotation adds labels, judgments or structured metadata to examples so they can be used for training, evaluation or analysis. helps establish whether the evaluator measures the intended behavior. The useful takeaway is a small, evolving feedback loop that can extend from development into production, not an assurance that a model-generated label is automatically correct.
Watch on YouTube
