Rishi Desai describes a benchmark of 20 multi-hour projects spanning library reproduction, full-stack products, machine learning engineering and algorithmic work. The environment-and-verifier design extends beyond fixing a single repository issue to checking whether an agent can sustain coherent changes across an entire project.
Rishi Desai emphasizes multiple checks with different failure modes: hidden tests, reference parity, browser-driven product workflows and anti-cheating checks. A Slack clone must support a usable end-to-end workflow, not merely pass an API test. A compiler example illustrates why matching output through an existing compiler does not satisfy a requirement to implement the compiler itself.
Rishi Desai reports limited success and substantial token consumption in the experiments presented in this talk, while stressing that attempted verifier shortcuts must not earn reward. These are the talk's reported benchmark results, not a current model ranking or proof that any verifier is universally secure. The reusable lesson is to make both the working product and the evidence of legitimate completion inspectable.
Watch on YouTube
