Stephanie Jarmak describes a coding-agent test in which a prompt change reduced cost but did not cause the agent to use the intended tool. She argues that a metric is useful only when it answers the actual decision question, and explains how changing the agent harnessAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. can change comparative results.
Jarmak recommends representative workloads, repeated runs and observability across context, tool calls and returned data. She contrasts public benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. with evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. designed for a specific product, and uses search precision and recall to show why a single score can obscure tradeoffs.
She discusses enforceable approval gatesAn agent permission boundary defines and enforces which resources and actions an automated agent is allowed to use., the capacity needed for human reviewHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences., and independent records for detecting tampered agent traces. Her closing example uses production data and shadow measurement to test whether changes improve real work.
Watch on YouTube
