Self-Improving Agents with Traces, Datasets and Evals

AI Engineer17m 7s
Marc Klingen against a black background beside the blue and white headline “AGENT IMPROVEMENT”.
2 VIEWS
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Marc Klingen explains how production tracesAn execution trace records the sequence of operations and results produced while a system carries out a task. become the evidence for improving an AI application. Instead of changing prompts by intuition alone, teams collect failures and feedbackAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior., curate representative datasetsAn evaluation set is a collection of examples kept for measuring an AI system rather than training it., compare experiments and deploy only after checking whether the proposed change improves the behaviorEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. they actually want.

    The demonstration uses a changelog agent to show how a coding agent can propose prompt and evaluation changes and test them against earlier cases. Klingen emphasizes human ownership of goals, dataset quality and deployment boundaries: an automatically improved score can otherwise reflect overfitting or reward hackingReward hacking occurs when a system improves a measured reward or score without achieving the intended underlying goal. rather than a better product.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools75
    • Models & learning25
    • Infrastructure & compute75
    • Research & evaluation100
    • Safety & security75
    • Business & productivity75
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Marc Klingen against a black background beside the blue and white headline “AGENT IMPROVEMENT”. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 6 October 2026 and duration 17m 7s.

    Marc Klingen presents an open-source agent improvement loop that turns production evidence into datasets, evaluations and experiments, while keeping goals and deployment decisions under human control.