Benchmark contamination undermines test independence by exposing information that should remain unavailable during evaluation. It can arise through training data, cached outputs, shared state, persistent tools or communication between runs.
Contamination does not automatically prove intentional manipulation. Researchers should identify the channel, measure its effect and rerun the evaluation under controlled conditions before drawing conclusions about capability.
ELI5
Benchmark contamination is like exam questions or answers leaking before the test. A strong score may then reflect prior access rather than genuine performance.
For example, if one agent writes an answer to a shared page and a later test run reads it, the later result has been contaminated by information outside the intended task.
What can cause benchmark contamination?
Causes include training-data overlap, answer caches, writable shared resources, retained session state and communication between evaluated runs.
How should researchers respond to suspected contamination?
They should preserve evidence, isolate the suspected channel, repeat the test and clearly disclose how the issue changes the interpretation of results.
