What is overfitting?

Definition

A system can perform well on familiar examples by exploiting incidental cues or adapting too closely to repeated feedback. This can affect model training as well as prompts, retrieval choices and an agent's evaluation workflow.

Held-out examples help assess whether the method works beyond the cases used to develop it. Reliable conclusions also need representative data and appropriate separation between tuning and final evaluation.

ELI5

Overfitting is doing well on the examples you practiced but not on genuinely new ones. The system may have learned a shortcut rather than the useful general rule.

For example, an assistant might rely on a company-name pattern that works in a small test list. Trying unseen companies can reveal whether that shortcut actually helps.

Can an agent workflow overfit without changing model weights?

Yes. Prompts, rules and evaluation procedures can be tuned too closely to a known set of examples.

Why use held-out cases?

They provide evidence about behavior on examples that did not guide the method's development.

  1. Two flat model diagrams feeding a probability gauge beside the words AI Models Test Market Signals
  2. Ryan Ahern against black beside the headline Don't Leak the Answer.
  3. Ryan Ahern against black beside the headline Test Yesterday's Web.
  4. Nick Heiner on the left against black beside the headline ‘BENCHMARKS VS REALITY’, blue comparison bars and converging lines.
Definition card for overfitting: What is overfitting?

Overfitting occurs when a model or tuned process learns patterns specific to its development examples that do not generalize reliably to new cases.