Rayan Krishnan describes a jagged frontier in which models rank differently across domains, repositories and workloads. He argues that model cards and token prices alone do not establish which system will be most effective for an organization's tasks.
Using finance and spreadsheet evaluation examples, Krishnan distinguishes cost per taskCost per completed task measures the total AI, tool, infrastructure, retry, and repair expense for each verified useful outcome. from input and output token prices. A model with a lower listed price can consume more tokens or perform worse on a particular task, while the best cost-performance choice can change across subsets of work.
Krishnan discusses choosing between partial-credit and fully correct answer metrics, and designing evaluations that are valid, reliable and able to distinguish models. He also highlights reward hackingReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal., infrastructure for long-running evaluations and the need to measure multi-turn work.
As a case study, Krishnan describes using prior software issues and their resolutionsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. to evaluate tools against an organization's own work. He presents that experience as a way to inform model selectionModel selection chooses the AI model whose capability, quality, cost, speed, safety, and operating constraints best fit a task., usage limits and routingAI model routing sends each request to a model chosen for that request's complexity, cost, speed, privacy, or other requirements. rather than assuming one model is best for every task.
Watch on YouTube
