Preference optimization trains a model from signals that compare outputs instead of requiring one exact target answer for every input. Feedback may rank two candidates, assign scores or indicate which behavior better satisfies quality and safety criteria.
The method is often used after a model already has the underlying capability. It can reduce recurring failure patterns, such as repetition, omitted content or degenerate audio, by making preferred outputs more likely relative to poor ones.
Preference data must represent the intended users and task. Biased, inconsistent or overly narrow judgments can shift the model toward superficial preferences or weaken useful behavior that the feedback did not measure.
ELI5
Preference optimization teaches a model by showing which of two answers is better. The model learns to make future outputs more like the preferred examples.
For example, reviewers can prefer a complete spoken sentence over one that repeats a phrase or skips words. The training then discourages those specific mistakes without teaching speech from the beginning.
