What is reinforcement learning from human feedback?

Definition

A common process asks people to compare candidate responses and uses those judgments to learn a reward signal. Reinforcement learning then adjusts the model toward behavior that receives a higher reward. The feedback represents the preferences and instructions captured by that process.

A preference-based reward is not identical to truth or success on every task. Results depend on the feedback, reward design and evaluation, and a model may learn patterns that satisfy raters without meeting a different deployment objective.

ELI5

Imagine coaching an assistant by showing which of several answers people prefer, then using those judgments to guide further practice.

For example, reviewers might favor clear, helpful explanations. The model still needs separate checks when a task requires exact factual or numerical correctness.

Is preferred output always correct?

No. Human preferences can reflect style or usefulness as well as accuracy, and the reward may miss important requirements.

Does it replace evaluation after training?

No. The resulting behavior still needs checks against the actual tasks and constraints of deployment.

  1. Diogo Almeida in a blue top on black beside the headline ‘BEYOND RLHF’ and blue branching decision lines.
  2. Ross Taylor and Chengxi Taylor in blue tops beside the blue and white headline Long-Horizon Agents on black.
Definition card for reinforcement learning from human feedback: What is reinforcement learning from human feedback?

Reinforcement learning from human feedback trains a model using signals derived from human judgments about its behavior or outputs.