Harbor Rollouts: Agent Evals, Production and Training

AI Engineer21m 11s
Alex Shaw and Ryan Marten against a black background beside the blue and white headline “AGENT ROLLOUTS EVAL TO TRAINING”.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Alex Shaw treats agent development as empirical machine learning rather than predictable conventional software. Model choice, instructions, tools and environments interact probabilistically, making repeated trials and relevant evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. essential. The session, credited by the publisher to Shaw and Ryan Marten, uses Harbor to give those trials a shared execution format.

    Harbor combines instructions, a sandboxed environmentAn AI agent sandbox is an isolated execution environment that limits which files, processes, networks, credentials, and external systems an agent can access. and a verifier, preserving the agent trajectoryAn execution trace records the sequence of operations and results produced while a system carries out a task. and the resulting reward. Shaw recommends building evaluations around real work before choosing a model, then reusing the same machinery for parallel production tasks. A map-and-reduce example extracts recurring human corrections from coding sessions to inform better tests.

    The resulting trajectories can also support supervised training, reinforcement learningReinforcement learning trains an AI system to choose actions using feedback about the results of its earlier choices. and iterative skill improvement. These applications share infrastructure, but their reliability still depends on task design, useful verification and attention to overfitting or reward hackingReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal.. Hiring invitations and benchmark-tour promotion are omitted from this account.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools25
    • Models & learning75
    • Infrastructure & compute50
    • Research & evaluation100
    • Safety & security25
    • Business & productivity25
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Alex Shaw and Ryan Marten against a black background beside the blue and white headline “AGENT ROLLOUTS EVAL TO TRAINING”. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 24 July 2026 and duration 21m 11s.

    In a session credited to Alex Shaw and Ryan Marten, Shaw explains how Harbor connects agent evaluation, production work and training through sandboxed tasks, preserved trajectories and verifiable outcomes.