Pierluca D'Oro introduces a replay agent that blindly repeats recorded successful actions. In the reported tests, this simple script can match or outperform the model that produced those actions on deterministic benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions.. The example illustrates how a benchmark may reward a repeatable sequence rather than robust interactionAn AI benchmark shortcut is a cue or exploitable pattern that raises a system's score without demonstrating the intended underlying capability., and why pass-at-k scores can inherit that weakness in deterministic environments.
Pierluca D'Oro proposes varying task data, interface appearance and starting states while verifying that every generated configuration remains valid. The described Android benchmark combines task templates, matching verifiers and mock data through a compiler-like process. The emphasis is on generating diverse, functioning tasks, since producing large amounts of application code alone does not ensure a trustworthy evaluation environment.
Pierluca D'Oro distinguishes randomness in an agent’s actions from variation in its environment. Confidence intervals that account only for repeated attempts at one fixed configuration can substantially understate uncertainty. The proposed evaluation methodEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. respects the benchmark’s hierarchical structure, helping reveal when an apparent difference between models is insufficient evidence for a deployment decision.
Pierluca D'Oro reports that frontier agents can be sensitive to changes such as interface themes and initial screensComputer use lets an AI agent interact with software through interface controls such as screens, buttons, keyboards and pointers., even when the underlying task is similar. The practical recommendation is to test this robustness explicitly, examine replayability and report uncertainty honestly. The presentation’s financial example illustrates the potential cost of a mistaken model choice; it is a hypothetical scenario rather than a measured deployment loss.
Watch on YouTube
