Original paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023).
Direct Preference Optimization (DPO) is an offline method for aligning a language model with pairwise preferences. Each training example contains a prompt, a preferred response, and a rejected response:
(prompt, chosen response, rejected response)
DPO increases the relative likelihood of the chosen response while keeping the policy close to a fixed reference model, usually the supervised fine-tuned model.
The usual KL-regularized RLHF objective is
\beta D_{\mathrm{KL}} \left( \pi(\cdot \mid x),|,\pi_{\mathrm{ref}}(\cdot \mid x) \right). $$
It asks the policy to maximize reward without moving too far from the reference policy. The optimal policy has the closed form
\frac{1}{Z(x)} \pi_{\mathrm{ref}}(y \mid x) \exp\left(\frac{r(x,y)}{\beta}\right). $$
Rearranging it gives
\beta \log \frac{\pi_r(y \mid x)} {\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x). $$
DPO inserts this implicit reward into the Bradley--Terry preference model. For a chosen response
\mathbb{E}{(x,y_w,y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \left[ \log \frac{\pi\theta(y_w \mid x)} {\pi_{\mathrm{ref}}(y_w \mid x)}
\log \frac{\pi_\theta(y_l \mid x)} {\pi_{\mathrm{ref}}(y_l \mid x)} \right] \right) \right]. $$
The expression inside the sigmoid is the difference between two implicit rewards. Training succeeds when the chosen response gains more probability over the reference than the rejected response does. The parameter
This is the central idea of DPO: the reward model in RLHF can be reparameterized through the policy, so a separate reward model and PPO optimization loop are not required. Training becomes a binary classification problem over response pairs. It is still different from SFT: SFT raises the likelihood of one target response, whereas DPO learns from the contrast between a chosen and rejected response.
DPO is most useful when the target behavior can be represented by a fixed dataset of response pairs. Typical examples include adjusting a chat model's helpfulness, tone, safety boundaries, or instruction-following style. Human or AI annotators only need to select the better response; they do not need to assign a calibrated score across many unrelated prompts.
DPO is a weaker fit when the model must continually improve on outputs outside the fixed dataset, when a reliable programmatic reward is available, or when new on-policy feedback can be collected during training. PPO, GRPO, or Online DPO can use fresh generations and may be better suited to reasoning tasks, verifiable objectives, or settings with substantial distribution shift.
- IPO (Identity Preference Optimization) replaces DPO's logistic construction with an identity-based objective and addresses failure modes such as overfitting on nearly separable preferences. A General Theoretical Paradigm to Understand Learning from Human Preferences (Azar et al., 2023).
- SimPO removes the reference model, uses length-normalized sequence log-probability as the implicit reward, and adds a target margin. SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al., 2024).
- Online DPO periodically samples new response pairs from the current policy and obtains preference labels from humans, a reward model, or an LLM judge. This reduces the mismatch between a changing policy and a fixed offline dataset, at the cost of repeated generation and labeling. Direct Language Model Alignment from Online AI Feedback (Guo et al., 2024).
- Cal-DPO adds absolute calibration to the contrastive objective. DPO mainly constrains the relative reward difference within a pair; Cal-DPO also makes the learned implicit rewards comparable in scale to the underlying rewards. Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (Xiao et al., 2024).
-
AlphaPO generalizes the usual logarithmic implicit reward with an
$\alpha$ -parameterized reward shape. The extra parameter provides finer control over likelihood displacement and over-optimization. AlphaPO: Reward Shape Matters for LLM Alignment (Gupta et al., 2025).
| Aspect | PPO-based RLHF | DPO |
|---|---|---|
| Feedback representation | A reward model converts feedback into a scalar reward | The policy is trained directly on chosen/rejected pairs |
| Training mode | Online or on-policy: generate responses during optimization | Offline: reuse a fixed preference dataset |
| Models commonly needed | Policy, reference model, reward model, and value model | Policy and reference model |
| Optimization target | Maximize estimated reward while controlling policy drift | Increase the chosen response's implicit reward over the rejected response's |
| Main strength | Can explore current policy outputs and optimize a scalar objective | Simple and relatively inexpensive preference fine-tuning |
| Main weakness | Complex and sensitive to reward/value estimation | Limited by the coverage and quality of offline pairs |
Many LLM alignment pipelines have moved from PPO-RLHF to DPO mainly because they use preference signals differently. In a conventional PPO-RLHF setup, a scalar reward model is first trained on pairwise preferences. The current policy then generates fresh samples and is further optimized with policy gradients using that reward. PPO is therefore a better fit when new generations can be scored reliably and ongoing online sampling and exploration matter. DPO, by contrast, updates the policy directly from chosen/rejected pairs. It dispenses with a separate reward model and an additional RL optimization stage, making it better suited to static, offline preference data. In practice, asking annotators which of two answers is better is also usually easier than asking for absolute scores that remain calibrated across examples. DPO can use those pairwise preferences directly, simplifying the training pipeline and reducing the risk of over-optimizing an explicit reward model.