A golden dataset contains representative cases that have been reviewed and annotated to a quality standard. Teams use it to compare models, prompts and judges against the same reference rather than relying on impressions from changing production traffic.
The dataset must evolve carefully as products and user behavior change. New examples should pass domain review, while versioning and provenance preserve which reference set produced each evaluation score.
ELI5
A golden dataset is a carefully reviewed set of examples with trusted labels or expected results. Teams use the same reference cases to compare AI models, prompts and evaluation methods fairly.
For example, experts can approve 500 support questions and correct answers, then test each new assistant version against them. The dataset should be versioned and updated carefully so scores remain connected to the exact reference used.
