Alvin Pane argues that teams should measure the cost, latency and outcomes of a taskCost per completed task measures the total AI, tool, infrastructure, retry, and repair expense for each verified useful outcome. rather than treating token consumptionAI token efficiency measures how effectively a model or workflow turns consumed input and output tokens into useful results. as a measure of success. He describes instrumenting model calls and comparing repeated runsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. of the same production system.
He discusses prompt cachingPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work., optimizing instructions and using smaller models for suitable work. He cautions that compression and model downsizing can introduce failures or overfitting.
Using a synthetic example, he explains shadow comparisons, manual review of regressions and improvementsAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked., paired confidence intervals and separate checks for critical categories. His numerical examples and results are attributed to the talk.
Watch on YouTube
