Frontier AI Models Are Losing Monitorability - Reilly Haskins

MTS18m 43s
Theo Jaffee in blue and Reilly Haskins in navy flank the blue and white headline 'CAN WE MONITOR AI?' on black.
1 VIEW
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Reilly Haskins describes METR's live per-action monitorAn AI monitoring agent observes systems or information sources, interprets signals, and reports or responds to relevant changes. in a discussion with Theo Jaffee. Before an evaluation agent executes a proposed action, another language model assesses it for possible real-world harm. Suspicious actions halt for human reviewAn agent permission boundary defines and enforces which resources and actions an automated agent is allowed to use., allowing a researcher to approve the action or stop the runHuman judgment in AI is the accountable interpretation and decision-making people contribute when setting goals, evaluating evidence, and managing consequences..

    Reilly Haskins argues that a monitor is only useful if it covers the relevant inference. Local models, external APIs and inconsistent logging can leave gaps. Structured transcripts and explicit role boundariesAn agent role boundary defines the specific responsibility, tools and decisions assigned to an AI agent. help resist prompt injection, but no known formatting choice eliminates jailbreaks or deceptive evasion. Red-teamingAdversarial evaluation deliberately searches for inputs and edge cases that reveal where an AI system's rules or behavior fail. and documenting a control system's assumptions remain essential.

    The discussion distinguishes increasingly opaque model reasoning from action-level oversight. Reilly Haskins notes that stronger models may reason without a readable chain of thought, but monitor models may also improve. Cybersecurity tasks, very difficult tasks and more capable agents raise evaluation risks; automated enforcement through a central inference platform can reduce reliance on researchers interpreting written policies correctly.

    Reilly Haskins discusses alert fatigue and the value of reviewers who understand a task's normal behavior. The current implementation's reported cost and latency overheads might be reduced with cheaper first-stage monitors, escalation hierarchies or less frequent checks, though these choices involve safety tradeoffs. He recommends decomposing monitoring claims into explicit subclaims and examining the evidence behind each one.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools75
    • Models & learning75
    • Infrastructure & compute75
    • Research & evaluation100
    • Safety & security100
    • Business & productivity25
    • Creative & media0
    • Robotics & perception0
    • Society & policy25

    Share this video

    Theo Jaffee in blue and Reilly Haskins in navy flank the blue and white headline 'CAN WE MONITOR AI?' on black. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 29 September 2026 and duration 18m 43s.

    Reilly Haskins explains how action-level monitoring can reduce risks during AI evaluations, while emphasizing uncertain coverage, adversarial evasion and human-review limits.