A large language model as a judge receives an example, rubric and evaluation prompt and produces a score or preference. It can automate open-ended quality checks that are difficult to express through exact deterministic metrics.
The judge can be biased or sensitive to prompt wording, so teams calibrate it against human-reviewed golden data. Transparent prompt comparisons and score monitoring help detect changes before automated judgments become a hidden source of quality drift.
Acronyms and aliases
LLM as a judge acronymLLM judge acronym
Related terms
Frequently asked questions
Why use a large language model as a judge?
It can evaluate open-ended outputs at scale and apply natural-language criteria more flexibly than simple exact-match metrics.
How can teams improve the reliability of a model judge?
They can calibrate it against expert-labeled examples, compare prompts transparently and monitor agreement and score changes over time.
