What is look-ahead bias?

Definition

Future outcomes, revised data or later documents can leak into a retrospective test and make predictions appear stronger than they actually were. In model-based systems, restricting retrieval is only one part of examining what information the system may already contain.

Evaluation should pin timestamps and distinguish prediction from analysis conditioned on known outcomes. If a test supplies the actual result as an input, its later reasoning cannot be counted as an advance prediction of that result.

ELI5

Look-ahead bias is accidentally letting a test see the answer before it makes its prediction. That makes the result seem more impressive than it should.

For example, testing an old business prediction with a current company page can reveal changes that happened afterward. The evaluation should separate what was known then from what became known later.

Can timestamped sources remove every leakage risk?

No. They help restrict retrieved material, but model knowledge, revised datasets and other supplied inputs may still contain later information.

Is analysis after an event the same as forecasting it?

No. Explaining consequences after receiving an event's actual result is a different task from predicting that result before it happened.

  1. Ryan Ahern against black beside the headline Test Yesterday's Web.
  2. Kristian Fagerlie against a black background with the blue-and-white headline "REPLAY THE MISS".
  3. Ryan Ahern against black beside the headline Don't Leak the Answer.
Definition card for look-ahead bias: What is look-ahead bias?

Look-ahead bias occurs when a historical evaluation uses information that would not have been available at the original decision time.