A large language model as a judge receives an example, rubric and evaluation prompt and produces a score or preference. It can automate open-ended quality checks that are difficult to express through exact deterministic metrics.
The judge can be biased or sensitive to prompt wording, so teams calibrate it against human-reviewed golden data. Transparent prompt comparisons and score monitoring help detect changes before automated judgments become a hidden source of quality drift.
ELI5
A large language model as a judge uses one language model to score, compare, or critique another system's output. It is useful when quality is hard to measure with an exact rule.
For example, a judge model can compare two support answers against a rubric for accuracy and clarity. People should first calibrate its scores against reviewed examples because the judge can have its own biases and prompt sensitivities.
Why use a large language model as a judge?
It can evaluate open-ended outputs at scale and apply natural-language criteria more flexibly than simple exact-match metrics.
How can teams improve the reliability of a model judge?
They can calibrate it against expert-labeled examples, compare prompts transparently and monitor agreement and score changes over time.
