Rustem Feyzkhanov argues that each organization needs benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. reflecting its own tools, policies and workflows. He contrasts production tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. with repeatable offline experiments that compare success, cost, latency and retries while holding the environment and evaluators steady.
Rustem Feyzkhanov describes executable tasks with realistic environments, oracle solutions and verifiers that examine final state, traces and artifacts. He covers benchmark CI, reward hackingReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal., changing one component at a timeAn ablation study measures how a system changes when a particular component or input is removed or disabled., expanding tests from production failures and keeping an unseen holdoutAn evaluation set is a collection of examples kept for measuring an AI system rather than training it.. Audience questions address common and edge-case coverage, an illustrative 80/20 split and expert review where evaluators disagree. Scale and performance claims are attributed to the speaker.
Watch on YouTube




