Ramakrishnan Lokanathan describes evaluationsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. as bounded signals of whether an AI product meets its promises. He recommends defining the intended job, prohibited outcomes and stopping conditions before designing tests, then examining intermediate steps as well as the final answer.
He advocates combining automated deployment checksDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state. with human reviewHuman-in-the-loop AI keeps a person involved in reviewing, correcting, approving, or guiding an AI system's work. and adversarial tests, and retaining regression casesAI regression testing reruns preserved cases to detect whether a model or agent update has broken behavior that previously worked. as models change. User reports and observed failures can become new testsAn AI feedback loop occurs when AI outputs or their consequences become inputs that influence later model or agent behavior., while a high score can indicate that an evaluation needs more challenging cases.
Lokanathan introduces eSlams, an open-source project that uses games such as chess and Go to explore sequences of decisions. He presents games as an additional evaluation perspective rather than proof that a winning model is better at every task.
Watch on YouTube
