Agent evaluation measures the behavior of a system that may make several decisions and tool calls before producing a result. It can assess final accuracy as well as the quality, safety, cost, and reliability of the path taken.
A useful evaluation defines the task and scoring rules before judging the run. Repeating the test across varied cases helps reveal whether an apparent improvement is dependable or limited to one example.
ELI5
Agent evaluation checks how well an AI agent performs a whole task. It looks beyond whether the final answer sounds good and examines the questions, facts, actions, safety, and effort used along the way.
For example, a shopping agent should lose credit if it recommends an unavailable product, ignores the budget, asks ten unnecessary questions, or uses a confusing format, even when its final paragraph reads smoothly.
What can an agent evaluation measure?
It can measure correctness, completion, consistency, tool use, cost, speed, recovery from errors, and compliance with permissions or instructions.
Why evaluate more than the final answer?
An agent may reach a plausible answer through unsafe or unreliable actions, so the process can matter as much as the final output.
