DeepSWE: A Contamination-Resistant Coding Benchmark - James Shi

AI Engineer17m 34s
James Shi in a blue top with two open hands beside the blue and white headline 'TEST ENGINEERING NOT LEAKS' on black.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    James Shi presents DeepSWE as a benchmarkA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. intended to test unfamiliar software-engineering work without relying on public pull-request solutions. He describes 113 newly authored tasksAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. across 91 repositories and five programming languages. Removing publicly available answers reduces one source of contaminationBenchmark contamination occurs when evaluation tasks, answers or related information become available to the AI system being tested., but does not make any benchmark a complete measure of engineering ability.

    Shi examines how evaluation results can hide weaknesses. Closely related tasks can overweight a narrow capability, while tests tied to helper names or a specific implementation can reject a valid solution. He contrasts exploratory and literal agent behavior and discusses cases where agents miss multipart requirements or find answers in retained Git history. The reported leaderboard reflects the presentation's evaluation setup, not a current universal ranking.

    The talk emphasizes tasks written by contributors who understand their repositories. Concise, high-level prompts require agents to explore existing conventions and coordinate larger changes without being handed a prescribed implementation. Shi argues that verifiers should test requested observable behaviorDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state. rather than a particular private helper structure.

    Shi also distinguishes underlying model capability from the prompts, tools and runtime surrounding it. A brief instruction suggesting that tests are already handled can discourage self-verification. A common harnessAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. enables a more controlled comparison, but differences from native agent products and the task mix remain limitations.

    For the benchmark's next iteration, Shi describes separating agent and evaluator environments, removing unnecessary Git references and improving standardized reports. Debugging, refactoring, broader repositories and less prescriptive evaluation remain areas for expansion. These changes are methodological proposals and reported work, not proof that benchmark scores guarantee production reliability.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows75
    • Coding & tools100
    • Models & learning50
    • Infrastructure & compute10
    • Research & evaluation100
    • Safety & security25
    • Business & productivity0
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    James Shi in a blue top with two open hands beside the blue and white headline 'TEST ENGINEERING NOT LEAKS' on black. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 26 July 2026 and duration 17m 34s.

    James Shi describes DeepSWE's original engineering tasks and behavior-oriented tests, explaining how answer leakage, grading assumptions and agent scaffolding complicate comparisons between coding models.