Vivek Trivedy describes a recurring improvement loop: collect traces from agents operating in an environment, identify useful patterns, and run experiments to test changes. Traces expose behavior that is difficult to predict from prompts and orchestration alone, including failed interactions, user feedback and degradation after context compaction.
Vivek Trivedy argues that trace analysis itself needs careful engineering because large trace collections can be expensive and individual runs can exceed a reviewer’s context window. He proposes treating long traces as queryable external data and choosing the least costly model that can perform the specific review task reliably, rather than using a frontier model for every step.
Vivek Trivedy describes harness changes and domain-specific fine-tuning as complementary approaches. He recommends first using fast feedback to improve the surrounding instructions and tools, then considering fine-tuning when further changes stop helping, and revisiting the harness afterward. Reported savings from legal trace-judging experiments are examples from his account, not a general cost guarantee.
Vivek Trivedy identifies three practical outputs of trace mining: curated examples for distillation or fine-tuning, evaluations and test environments, and concise material for human reviewers. He emphasizes richer feedback than a single pass-or-fail score, while acknowledging that agents optimizing a metric can exploit weaknesses in how it is measured.
Vivek Trivedy connects this process to continual learning across training data, harness design and memory. He argues that long-lived agents need mechanisms for revising useful knowledge over time instead of endlessly appending information. The talk presents this as an evolving research direction, with successful updates requiring evidence that they improve the tasks users actually care about.
Watch on YouTube
