Tisha Chawla and Susheem Koul explain why ordinary reruns are a poor way to debug production agents. Models, tools and external data can all change between attempts, so the same input may follow a different path and hide the failure that engineers need to inspect.
Their alternative is replayability rather than bit-for-bit determinism. A production run records the inputs and outputs at every node, along with model versions and relevant state. Engineers can later replay that trace locally, replace expensive or unstable boundaries with mocks and isolate the exact step that produced the bad result.
Testing then combines deterministic assertions with behavioral evaluation. Exact checks cover stable facts such as tool names and arguments, while evaluators judge whether a response still meets the intended outcome. This gives teams a repeatable debugging path without pretending that every valid agent run must produce identical text.
Watch on YouTube
