What is a task horizon?

Definition

One way to measure it is to group tasks by how long skilled people typically take to complete them, then estimate how the system's success rate changes with task length. The measure depends on the tasks, allowed tools and the definition of success.

A longer measured horizon can indicate greater reliability on extended work. Scores may change when evaluators tighten checks on whether a result was actually achieved.

ELI5

For example, if an assistant completes most of the tasks that take a person ten minutes but fails most hour-long tasks, an evaluation can estimate where its success starts to fall.

That estimate is about the tested work. It does not tell you how long the assistant can work independently on every subject.

Is a task horizon measured by how long the model runs?

In this evaluation approach, task length comes from a human completion-time reference. The model's own runtime is a separate measure.

Why specify a success rate?

Different thresholds produce different horizons. Specifying the threshold makes the result interpretable and easier to compare.

  1. Theo Browne beside blue and white text reading GPT-5.6 Preview.
Definition card for task horizon: What is a task horizon?

A task horizon estimates the length of tasks an AI system can complete at a specified success rate.