From Agent Traces to Agent Simulations

AI Engineer20m 24s
Rustem Feyzkhanov with one hand resting at his chin beside the blue and white headline TEST THE WHOLE AGENT on a black background.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Rustem Feyzkhanov argues that each organization needs benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. reflecting its own tools, policies and workflows. He contrasts production tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. with repeatable offline experiments that compare success, cost, latency and retries while holding the environment and evaluators steady.

    Rustem Feyzkhanov describes executable tasks with realistic environments, oracle solutions and verifiers that examine final state, traces and artifacts. He covers benchmark CI, reward hackingReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal., changing one component at a timeAn ablation study measures how a system changes when a particular component or input is removed or disabled., expanding tests from production failures and keeping an unseen holdoutAn evaluation set is a collection of examples kept for measuring an AI system rather than training it.. Audience questions address common and edge-case coverage, an illustrative 80/20 split and expert review where evaluators disagree. Scale and performance claims are attributed to the speaker.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools50
    • Models & learning10
    • Infrastructure & compute25
    • Research & evaluation100
    • Safety & security25
    • Business & productivity25
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Rustem Feyzkhanov with one hand resting at his chin beside the blue and white headline TEST THE WHOLE AGENT on a black background. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 25 July 2026 and duration 20m 24s.

    Rustem Feyzkhanov explains how to turn production traces into repeatable simulations that test an AI agent's entire stack, not just its model.