Air Canada’s Nathanael Weill describes an earlier retrieval projectRetrieval is the process of selecting relevant stored information and returning it to an AI system for the current task. where generated questions with known source pages enabled faster automated evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. than scarce expert review alone. He says the team used the resulting feedback to simplify and improve its system.
For contract-review agents, he discusses generating cases from rules and templates, removing clauses, and checking consistency under irrelevant changes. He cautions that synthetic tests help development but are not sufficient on their own to decide whether to ship.
Watch on YouTube
