Theo Browne discusses Sonnet 5 and the reported status of Fable 5 in a July recording, before examining published benchmark results. He questions benchmark contaminationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and compares total task costsCost per completed task measures the total AI, tool, infrastructure, retry, and repair expense for each verified useful outcome., noting that lower token prices do not necessarily mean lower costs for an extended task.
Theo Browne describes his own attempts to recreate a web gameCode generation uses AI or another automated system to create source code from instructions, examples, schemas, or higher-level specifications. using several models. He reports that Sonnet used multiple subagents but took much longer than another run and produced a flawed result. His comparison uses a limited set of examples and does not establish universal model rankings.
The presenter considers when smaller agents might help a stronger orchestratorAI agent delegation assigns a bounded task, context, authority, and expected result from one participant to an agent or another agent., along with reasoning settings, refusal behavior and a reported output issue. His claims about security, pricing and product availability are time-specific and should not be treated as independent guarantees.
Watch on YouTube
