Kobie Crawford presents Snorkel’s approach to checking whether agent tasks are achievable, non-trivial, functionally correct and supported by reliable environmentsAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction..
He compares accepted and rejected tasks, explaining why failures caused by broken environments can obscure whether a model actually lacks a capabilityEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria..
Crawford describes a controlled training comparison in which the higher-quality task set produced a larger improvement under the same compute budget. He discusses underspecified tests, harder-to-verify tasks and human agreement in audience questions.
Watch on YouTube
