Ryan Stevens describes breaking bookkeeping into tasks and working with accountants to define success before building an agent evaluationEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria..
His team created synthetic businesses and accounting data to replay tasks, evaluate final results and compare accuracy with all-or-nothing success measures.
He discusses model selection, failure analysis, testing the contribution of skillsAn AI agent skill is a reusable package of instructions, resources, and tool guidance for performing a bounded kind of work. and memory, and recurring regression checks. He presents the observed results as findings from Ramp’s own experiments.
Watch on YouTube
