Benchmark variance shows whether a reported result is stable or sensitive to sampling and evaluation choices. Generative systems can produce sharply different outputs for the same prompt, so one result may not represent typical quality.
Teams can estimate variance with repeated runs, broader prompt sets, multiple judges, confidence intervals, and preserved settings. Direct workflow tests reveal whether the model is reliable for the creator's intended shots rather than merely strong on average.
ELI5
Benchmark variance describes how much a model's test score might move if the test were repeated or used a different small sample. High variance means the reported number is less stable.
For example, on a ten-task set, one changed result moves the score by ten percentage points. Repeating the test and adding more representative tasks gives a clearer view of whether the apparent lead is dependable.
What can cause benchmark variance?
Random generation, prompt samples, model settings, judge differences, hardware, curation, small test sets, and changing model versions can cause it.
How can benchmark variance be reported?
Reports can include repeated runs, distributions, confidence intervals, failure rates, sample counts, settings, and representative outputs.
