Bertrand Charpentier compares public leaderboards and internal benchmarks, explaining how rankings can change across tasks, datasets and metrics. He illustrates how small samples and individual preferences can bias manual evaluation.
Bertrand Charpentier recommends evaluating representative examples with multiple metricsEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and considering latency and cost alongside qualityCost per completed task measures the total AI, tool, infrastructure, retry, and repair expense for each verified useful outcome.. He uses Pareto frontiers to show that different models can be suitable under different constraints, and discusses compression approaches in the question period.
Watch on YouTube
