What is an evaluation metric?

Definition

An artificial intelligence evaluation metric converts an aspect of system behavior into a value that can be compared across outputs, models or versions. Metrics can measure accuracy, omission rates, unsupported claims, task completion, safety violations, latency, cost and many other properties.

No single metric captures every important quality. A high aggregate accuracy score can hide rare but consequential failures, so teams often combine several metrics with case-level inspection and expert judgment that reflects the real cost of different errors.

Acronyms and aliases

AI eval metric variantAI evaluation metric variantartificial intelligence evaluation metric variant

Frequently asked questions

Is accuracy enough to evaluate an AI system?

Usually not. Accuracy can hide the type, severity and context of failures, especially when uncommon errors have serious consequences.

How should AI evaluation metrics be chosen?

They should be chosen from the intended task, user needs, failure modes and relative consequences of false, missing or unsafe outputs.

Videos explaining evaluation metric