Tisha Chawla and Susheem Koul explain why ordinary reruns are a poor way to debugDebugging is the process of finding why software behaved incorrectly and checking that a correction addresses the cause. production agents. Models, tools and external data can all change between attempts, so the same input may follow a different path and hide the failure that engineers need to inspect.
Their alternative is replayability rather than bit-for-bit determinism. A production run records the inputs and outputs at every node, along with model versions and relevant state. Engineers can later replay that traceAn execution trace records the sequence of operations and results produced while a system carries out a task. locally, replace expensive or unstable boundaries with mocksA test double is a substitute for a real component used to control or observe behavior during a test. and isolate the exact step that produced the bad result.
Testing then combines deterministic assertionsDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state. with behavioral evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. Exact checks cover stable facts such as tool names and arguments, while evaluators judge whether a response still meets the intended outcome. This gives teams a repeatable debugging path without pretending that every valid agent run must produce identical text.
Watch on YouTube
