Large language model judge calibration compares automated judgments with labels in a golden dataset. Teams revise the judge prompt, rubric or scoring method, then inspect whether agreement improves across representative and difficult cases.
Calibration is not permanent. Model changes, prompt edits and new product behavior can alter scores, so the process should be versioned, repeated and monitored with visible examples and reviewer ownership.
ELI5
Large language model judge calibration checks whether an AI evaluator scores work the way trusted human reviewers intend. The team compares its decisions with carefully reviewed examples and adjusts the instructions or scoring rules when the judgments do not match.
For example, people can score one hundred support answers for accuracy and helpfulness, then ask the AI judge to score the same answers. If the AI is too generous about incorrect facts, the rubric can be clarified and tested again on fresh examples.
How is a large language model judge calibrated?
Its scores are compared with trusted human annotations, and the prompt or rubric is revised until agreement is acceptable for the intended use.
Why does judge calibration need to be repeated?
Models, prompts and product traffic change, so a judge that matched earlier examples may drift from the current quality standard.
