Large language model judge calibration compares automated judgments with labels in a golden dataset. Teams revise the judge prompt, rubric or scoring method, then inspect whether agreement improves across representative and difficult cases.
Calibration is not permanent. Model changes, prompt edits and new product behavior can alter scores, so the process should be versioned, repeated and monitored with visible examples and reviewer ownership.
ELI5
Large language model judge calibration checks whether an AI evaluator scores work the way trusted human reviewers intend. The team compares its decisions with carefully reviewed examples and adjusts the instructions or scoring rules when the judgments do not match.
For example, people can score one hundred support answers for accuracy and helpfulness, then ask the AI judge to score the same answers. If the AI is too generous about incorrect facts, the rubric can be clarified and tested again on fresh examples.
