An AI benchmark shortcut can come from leaked answers, repeated templates, metadata artifacts, weak verifiers, narrow input patterns, or evaluation rules that reward an unintended strategy. A model may learn the shortcut during training or discover it through repeated interaction with the benchmark.
Researchers reduce shortcuts with fresh tasks, contamination checks, adversarial review, hidden test cases, robust verifiers, varied environments, and analysis of agent trajectories. The goal is to make success depend on solving the real problem rather than recognizing the test.
ELI5
A benchmark shortcut is a trick that helps an AI system score well without showing the ability the test was meant to measure. The system may notice an accidental clue in the questions or exploit a weakness in the scoring rules.
For example, an image test might accidentally give every cat photo a similar filename. A model that reads the filename could get a high score without learning to recognize cats in the image itself.
