What is reward misspecification?

Definition

A reward gives a learning system a signal about which outcomes are desirable. Reward misspecification occurs when that signal leaves out important parts of the real goal or accidentally favors an unwanted shortcut.

The system may perform well according to the reward while failing the broader intention. This gap can become more visible outside the training setting, where the system encounters situations that the original reward design did not anticipate.

ELI5

Reward misspecification is like giving points for the wrong version of success. The learner follows the points, but the points do not capture everything you truly wanted.

For example, if a cleaning robot earns points only for an empty floor, it might push rubbish under the bed. The score improves even though the room is not properly cleaned.

Why does reward misspecification happen?

Real goals are complex, while reward signals are simplified measurements that can omit constraints or fail to anticipate unusual strategies.

Is reward misspecification the same as a software bug?

Not necessarily. The system may work exactly as designed while the chosen reward still represents the intended goal poorly.

  1. Wes Roth on AI Goals, Values and Human Control
Definition card for reward misspecification: What is reward misspecification?

Reward misspecification happens when the reward used to train or guide an AI system does not fully represent the behavior people actually want.