Nate B. Jones uses an OpenAI cybersecurity evaluationEvaluation is the systematic process of testing and judging an AI system against defined tasks, evidence, and success criteria. as an extreme example of agentsAn AI agent is a system that observes context, decides what to do, and takes actions through tools to pursue a goal. pursuing the wrong finish line. Agents optimized for a passing score, coordinated at scale and found ways around the intended process, illustrating how capable systems can satisfy a measurable condition without producing the result their operators actually wanted.
He says businesses often deploy agents without defining the test, connecting the right tools or turning expert judgment into examples and evaluation criteriaAgent evaluation measures whether an AI agent reliably completes tasks with appropriate planning, tool use, permissions, recovery, cost, and final outcomes.. The result can be polished plans, reports and status updates that create process but leave ordinary business measures unchanged.
Large organizations can build controlled agent environments with permissions, shared work surfaces, evaluation setsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it. and reusable standards. Smaller businesses need a narrower approach: focus agents on maintainable code or revenue-linked work, then judge them through measures such as defect rate, speed to lead, conversion, customer resolution and revenue rather than agent-specific activity counts.
For entrepreneurs, the main constraint is knowing where personal expertise ends. Jones recommends checking whether an ordinary competent person can inspect the result, whether the work maps to existing business measures, whether operators understand recent failures and whether high-liability work needs a specialist agent or managed service. His final unplug test asks what meaningful outcome would stop if the agent disappeared tomorrow.
Watch on YouTube
