The set should represent the tasks and difficulty levels that matter for the intended use. Clear expected outcomes or scoring rules let teams compare models and versions on the same cases.
Small sets can be fast and informative but produce uncertain estimates. They are also easier to overfit through repeated tuning, so teams should preserve unseen cases and report sample size.
ELI5
An evaluation set is a collection of examples used to check how well an AI system performs. Each example includes an expected result or a clear rule for judging the answer.
For example, a document tool can be tested on invoices with different layouts, languages, and missing fields. The team should avoid tuning the tool on every test example, or the score may stop reflecting unfamiliar real cases.
