Versioned Web Search for Reliable Agent Backtesting

AI Engineer19m 12s
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Video summary

    Ryan Ahern introduces Exa Snapshot as a way to retrieve a known page or execute a search against a past point in time. He connects this to agent evaluation: searching today's web can reveal an acquisition or other outcome that a historical prediction was supposed to anticipate. Restricting retrieved evidence helps separate genuine predictive signals from information that arrived later.

    Ryan Ahern describes three patterns: enriching an existing model with historical unstructured data, adding an agentic research feature to a conventional model, and testing a fully agentic prediction system. Financial examples illustrate the evaluation problem rather than establish a profitable trading method. Product coverage and performance claims are attributed to the presenter, not independently validated here.

    Ryan Ahern uses an earlier workshop's company-name shortcut to show that historical retrieval alone does not prevent overfitting. Teams still need held-out examples and checks for generalization. The short talk is distinct from the separate hands-on workshop: it organizes the use cases and explains why the information available at prediction time matters.

    Original YouTube thumbnailWatch on YouTube

    Share this page

    Ryan Ahern against black beside the headline Test Yesterday's Web. Framed in blue with WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 19 October 2026 and duration 19m 12s.

    Ryan Ahern explains how point-in-time web search can constrain an agent's evidence during backtests, while holdouts remain essential to detect overfitting.