Memory Harnesses for Long-Running Research Agents - Stefania Druga, Sakana AI

AI Engineer13m 4s
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Video summary

    Stefania Druga treats agent memory as a write, manage and read control loop rather than a database alone. Her harness keeps a core trace, tests several recall policies and uses an oracle that supplies ground-truth memory without forcing the model to use it.

    On a literature-review task that fit within the model context, memory did not improve performance and only added cost. On XBench questions whose answers fell hundreds of steps outside the active context window, ranked recall improved answer accuracy over no memory and simple gating.

    The best-performing setup ranked a decision ledger, and the result held across both tested local models and an additional benchmark. The practical lesson is to measure recall policy as a first-class system metric because irrelevant memory consumes tokens and can send an agent in the wrong direction.

    Local execution provided control over data, traces, compute and evaluation, but serial inference made experiments slow. The talk closes by placing these tradeoffs within the wider goal of sovereign AI.

    Original YouTube thumbnailWatch on YouTube