Will Brown examines reinforcement learning when success cannot be checked with a simple rule, as it often can for a numerical answer, passing code tests or an expected database state. He describes an agent as a model plus a harness, interacting with tasks, tools and a world that produces feedback. Reports, purchases and customer interactions introduce less clearly defined objectives, so the challenge becomes constructing useful rewards without confusing an imperfect proxy with the outcome people actually want.
Will Brown proposes grounding that process in material from real deployments, including agent traces, documents and repositories. Comparing performance with and without relevant source material can expose a capability gap that supplies learning signal. For documents, models can generate and check answerable questions before removing the initial retrieval step; for code, completed changes and tests can be broken into tasks whose end states are known to be reachable. This backward construction verifies an easier problem and then trains the agent to recover the harder solution.
Will Brown describes controllable simulators as a way to recreate messy tool and web interactions when the production system's backend cannot be directly manipulated. Production traces help refine the simulation, while control over its state makes it possible to construct solvable tasks. Model judges and additional search can then inspect failures in hindsight, turn observations into rubrics and target those failure modes with new tasks. He emphasizes difficulty calibration and reward-hacking review: an agent may learn to increase its score without satisfying the intended objective.
Will Brown argues that environment design must also be tested through small training runs, because some weaknesses emerge only after optimization changes the agent's behavior. Tool-call metrics and trace review help expose those changes, while humans retain responsibility for defining the important goals and judgments. He also distinguishes refining skills through reinforcement learning from acquiring new knowledge, discussing supervised signals from the environment as a complementary route. Continual improvement in deployment is presented as a goal requiring monitored, traceable experiments, not a guarantee that these methods already solve every real-world task.
Watch on YouTube
