The presenter argues that modern coding modelsAn AI coding agent is a tool-using AI system that can inspect, modify, and validate software within a repository. have become difficult to compare with a single benchmarkA benchmark is a defined set of tasks, conditions, and scoring rules used to compare AI systems or measure progress.. Public demos can look impressive, yet repeated use reveals recognizable visual habits, and every evaluatorEvaluation is the systematic process of testing and judging an AI system against defined tasks, evidence, and success criteria. ends up relying on a personal collection of tasksWorkflow-specific AI evaluation tests models using the tasks, tools, constraints, and outcomes of a real working process. rather than one dependable measure of quality.
To make the comparison concrete, the presenter spends about $200 running the same broad app-building prompt through Fable 5.1 and Opus 5. The requested projects include reading and computer-skills apps for children, a piano improvisation tutor and an AI-oriented product. The generated results expose both surprising capability and obvious mistakes, including broken interface behavior.
A second test asks each model to create short promotional videos. Fable 5.1 produces a much stronger result for one product but a less convincing result for another, while the presenter also acknowledges that the vague promptsPrompt engineering is the practice of designing and refining instructions, context, examples, and constraints to obtain useful AI outputs. make the test subjective. The conclusion is not a definitive winner, but a practical warning that model quality now depends heavily on the task, prompt and evaluator's standards.
Watch on YouTube
