Doneyli De Jesus recounts work with a bank to evaluate an AI application against internal data, security constraints and compliance requirements. The proposed process uses a golden datasetAn evaluation set is a collection of examples kept for measuring an AI system rather than training it., traces, dimension-specific scores and thresholdsAn AI evaluation metric is a defined measurement used to quantify a particular aspect of an AI system's performance. before deployment.
Doneyli De Jesus describes adding a calculation tool to address numerical errors and combining deterministic checksDeterministic software verification checks an output with a repeatable procedure that produces the same result for the same inputs and state., path evaluation and model-based judgmentsA model-based grader uses an AI model to evaluate, compare or score outputs according to defined criteria.. Domain experts review failuresHuman-in-the-loop AI keeps a person involved in reviewing, correcting, approving, or guiding an AI system's work. and contribute feedback to future evaluation datasets.
Watch on YouTube
