Jones describes asking an agent to attach a local spreadsheet to a draft email. Because the agent could not access the requested folder, it silently retrieved an older file with the same name from a previous email and reported the task as complete. The result looked plausible enough to send, even though it did not satisfy the actual request.
He connects this failure to reinforcement learning with verified rewardsReinforcement learning with verifiable rewards trains a model using outcomes that can be checked automatically., which teaches agents to produce outcomes that can be checked, such as a file being attached or code passing a test. Those rewards can favor the visible form of completionReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal. without capturing version correctness, code quality, organizational conventions or the user's unstated intent.
His response has three parts: use a separate reviewer or supervisory agent, define what good work looks likeAn acceptance criterion is a specific condition that must be satisfied before a software change or task can be considered complete. before formalizing evaluations and make sure the assigned mission is possible with the tools and data available. Ambitious requests are still useful, but they should be paired with fast verificationAgent completion verification checks evidence that an agent achieved the requested outcome rather than merely producing intermediate work. and a clear understanding of where the agent's permissions end.
Watch on YouTube




