Backtest Agent Predictions Without Leaking the Answer

AI Engineer26m 41s
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Video summary

    Ryan Ahern demonstrates point-in-time retrieval and search with Exa Snapshot, then turns the idea into an acquisition-prediction exercise. Participants define yes-or-no features and test them against companies before an outcome was announced. The methodological aim is to keep future web information out of a historical evaluation, not to promise reliable financial predictions.

    Ryan Ahern explains why an article's displayed publication date is not always a reliable historical boundary: pages can be revised. He distinguishes gathering historical features for an existing model from testing a research agent or an entirely agentic workflow. A time-bounded retrieval tool controls one source of leakage, not all knowledge the model might carry.

    Ryan Ahern combines leadership-change and sale-exploration signals with AND and OR rules, showing the trade-off between finding acquisitions and raising false alarms. The winning workshop entries then expose a deliberate weakness in the exercise: without a holdout, participants can memorize company names. That failure is the practical lesson. The longer workshop contains this hands-on demonstration and is not a duplicate of the separate short overview.

    Original YouTube thumbnailWatch on YouTube

    Share this page

    Ryan Ahern against black beside the headline Don't Leak the Answer. Framed in blue with WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 19 October 2026 and duration 26m 41s.

    Ryan Ahern's hands-on workshop shows how historical search controls retrieved evidence but does not replace holdout sets or checks against memorizing the evaluation data.