Reward-based fine-tuning updates a pretrained model according to feedback expressed as a numerical reward. The reward may come from people, automated checks, a learned reward model or task-specific measurements, allowing training to favor outputs that better meet an objective.
The method can improve generation or decision-making, but the reward must represent the real goal accurately. A model may exploit shortcuts in an incomplete metric, and optimization can reduce other useful behavior. Careful evaluation, regularization and diverse feedback help manage those risks.
ELI5
Reward-based fine-tuning improves a pretrained model by scoring its behavior and updating it to make higher-scoring choices more likely. The scores can come from people, automatic checks, another learned model, or measurements from the task.
For example, a coding model can receive a higher reward when generated code passes tests and follows security rules. If the score overlooks maintainability, the model may learn shortcuts, so training needs varied feedback and broader evaluation.
How is reward-based fine-tuning different from supervised fine-tuning?
Supervised fine-tuning learns from target examples. Reward-based fine-tuning receives a score for generated behavior and updates the model to increase expected reward.
Can diffusion models use reward-based fine-tuning?
Yes. A reward can score properties of generated images, video or other samples, and optimization can adapt the diffusion model toward higher-scoring outputs.
