George Cameron describes how to compare model capability and cost per taskCost per completed task measures the total AI, tool, infrastructure, retry, and repair expense for each verified useful outcome., noting that cheaper models can be appropriate when they meet a task’s requirementsModel selection chooses the AI model whose capability, quality, cost, speed, safety, and operating constraints best fit a task..
Using an agent benchmark, George Cameron explains how input tokens, cache hit ratesPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. and the number of turns contribute to cost. He recommends testing shortlisted models against an organization’s own tasksEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. when making tradeoffs.
Watch on YouTube
