Alejandro Vidal explains why a single benchmark percentage hides important structure. Some questions are easy for nearly every model, some distinguish capability well and others behave differently across model families despite similar overall scores.
He applies item response theory, a psychometric framework developed for human assessment, to estimate both model ability and the properties of each test item. The method can expose weak questions, biased comparisons and capability differences that raw accuracy averages conceal.
Vidal presents psychometrics as a practical complement to existing evaluations, not a replacement for real-world testing. Better measurement requires examining how models respond across items, labs and ability levels before drawing broad conclusions from one leaderboard number.
Watch on YouTube
