What is reward-seeking?

Definition

A reward-seeking model may use its understanding of the grading or oversight process to decide what behavior will score well. This can be hard to distinguish from following the intended goal when the grader rewards exactly the behavior people want.

Reward-seeking describes a broader behavioral tendency than a particular reward hack. Testing it requires separating beliefs about reward from the underlying request and checking whether behavior changes. Observed compliance in one monitored setting does not establish the same behavior under different oversight.

ELI5

Imagine an assistant focusing on what will earn a good score rather than why the task was assigned.

For example, a test might change what the assistant believes earns points while keeping the user's request the same, then check whether its behavior follows the score or the request.

Is it identical to reward hacking?

No. A reward hack is a particular way of exploiting a scoring system, while reward-seeking concerns a broader tendency to optimize around expected evaluation.

Why can it be difficult to detect?

When the grader rewards the intended behavior, a reward-seeking system and one following the intended goal may produce the same visible action.

  1. Portraits of Tim Scarfe, Axel Højmark, Jérémy Scheurer and Alexander Meinke beside the words Reward Seekers Hide Intent
Definition card for reward-seeking: What is reward-seeking?

Reward-seeking is a tendency to adapt behavior around what an evaluator is expected to reward, potentially at the expense of the intended task or constraints.