Direct Preference Optimization
A preference-optimization training regime that teaches a model from preferred versus rejected answer pairs without fitting a separate reward model and PPO loop first.
Direct Preference Optimization, usually shortened to DPO, teaches a model from prompt-plus-choice comparisons: for each prompt, the model sees one answer people preferred and one they rejected, then learns to score the preferred answer higher without first building a separate reward model and reinforcement-learning loop.
At a glance
Released
May 2023
Authors
Rafael Rafailov, Archit Sharma, Eric Mitchell, et al.
Regime type
Training Alignment
Related modules
No related modules listed yet.
What It Is
Direct Preference Optimization is a post-training regime for alignment. Instead of asking annotators for one perfect answer, it uses side-by-side preference pairs: a prompt, a preferred response, and a rejected response. The model is then updated so the preferred response becomes more likely than the rejected one.Why It Exists
Classic reinforcement learning from human feedback, often shortened to RLHF, usually splits the job into two stages: fit a reward model from human rankings, then run reinforcement learning such as Proximal Policy Optimization, or PPO, against that reward. DPO exists because many teams want the behavior-shaping value of preference data without carrying that extra reward-model and reinforcement-learning machinery.How It Works
A base or instruction-tuned model is kept as the starting point. For each prompt, the training set provides two candidate answers: one preferred and one rejected. DPO compares how much the current model likes those two answers relative to a reference model, then directly nudges the current model toward the preferred answer and away from the rejected one. In practice, the preference comparison stands in for the separate reward-model-fitting step and the later PPO policy-update loop.Direct Preference Optimization training flow
Prompt + preferred / rejected pairDirect preference objectiveUpdated aligned modelPrompt + preferred / rejected pair to Direct preference objectiveDirect preference objective to Updated aligned model
Request and weight flow
\\text{preferred behavior gain} \\approx \\log \\pi_\\theta(y^+ \\mid x) - \\log \\pi_\\theta(y^- \\mid x) - \\log \\pi_{\\text{ref}}(y^+ \\mid x) + \\log \\pi_{\\text{ref}}(y^- \\mid x)