Sonam Pankaj argues that agent evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and traces often remain disconnected from the context retrieved on later runsRetrieval is the process of selecting relevant stored information and returning it to an AI system for the current task.. Her proposed runtime memory system combines semantic similarity with a utility score informed by previous outcomes.
She describes storing actionable reasoning rather than just user preferences and discusses turning recurring lessons into reusable skillsAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work.. She presents benchmark results from her team and notes risks including cold starts, noisy feedback and utility drift.
In a product-search demonstration, she marks a failed query and supplies feedbackAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior.; a later run retrieves that experience and broadens the search. The demonstration illustrates the proposed mechanism, while the reported benchmark gains are the speaker’s claims.
Watch on YouTube
