Pierluca D'Oro introduces a replay agent that blindly repeats recorded successful actions. In the reported tests, this simple script can match or outperform the model that produced those actions on deterministic benchmarks. The example illustrates how a benchmark may reward a repeatable sequence rather than robust interaction, and why pass-at-k scores can inherit that weakness in deterministic environments.
Pierluca D'Oro proposes varying task data, interface appearance and starting states while verifying that every generated configuration remains valid. The described Android benchmark combines task templates, matching verifiers and mock data through a compiler-like process. The emphasis is on generating diverse, functioning tasks, since producing large amounts of application code alone does not ensure a trustworthy evaluation environment.
Pierluca D'Oro distinguishes randomness in an agent’s actions from variation in its environment. Confidence intervals that account only for repeated attempts at one fixed configuration can substantially understate uncertainty. The proposed evaluation method respects the benchmark’s hierarchical structure, helping reveal when an apparent difference between models is insufficient evidence for a deployment decision.
Pierluca D'Oro reports that frontier agents can be sensitive to changes such as interface themes and initial screens, even when the underlying task is similar. The practical recommendation is to test this robustness explicitly, examine replayability and report uncertainty honestly. The presentation’s financial example illustrates the potential cost of a mistaken model choice; it is a hypothetical scenario rather than a measured deployment loss.
Watch on YouTube
