Stable Preference Optimization

• Stable preference optimization is a family of methods that modify training dynamics to improve alignment stability by addressing issues like high-gradient variance and off-policy bias.
• It adapts pairwise preference learning through techniques such as adaptive reference updates, timestep-aware sampling, and importance weighting to stabilize training.
• Empirical results in diffusion, language models, medical segmentation, and combinatorial optimization demonstrate significant improvements in alignment metrics and reduced training collapses.

Stable preference optimization denotes a line of preference-based alignment methods that target stable optimization dynamics rather than treating pairwise preference loss as sufficient on its own. In recent work, the term is used across text-to-image and video diffusion, large language and reasoning models, medical image segmentation, and neural combinatorial optimization. The recurring motivation is that vanilla Direct Preference Optimization (DPO) or closely related objectives can be brittle: they may rely on a frozen reference model, treat all preference pairs uniformly, suffer from off-policy bias, exhibit high gradient variance, or improve relative preference margins while failing to improve the absolute probability of the preferred output (Kang et al., 24 May 2025, Jian et al., 10 Jul 2025, Wu et al., 1 Feb 2026, Zhu et al., 6 Oct 2025, 2505.21893). Stable preference optimization therefore appears less as a single algorithm than as a family of training procedures that modify reference handling, sampling, weighting, supervision, or estimator construction to preserve alignment signal while reducing collapse, overfitting, and noisy updates (Nam et al., 17 Dec 2025, Najafi et al., 27 Oct 2025, Jiang et al., 2023, Liao et al., 10 Mar 2025).

1. Formal foundations and recurring objective structure

The common formal reference point is DPO. In the standard language-model setup, the reward parameterization is written as

$$r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\mu(y \mid x)},$$

with objective

$$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$

This formulation directly compares preferred and rejected responses relative to a reference policy and removes the explicit reward model used in RLHF-style pipelines (Wu et al., 1 Feb 2026).

In diffusion models, the same preference-learning logic is adapted to denoising trajectories. "Rethinking Direct Preference Optimization in Diffusion Models" defines a tractable Diffusion-DPO loss over preferred and non-preferred images, with timestep-wise implicit rewards computed from the difference between the trainable model’s and the reference model’s denoising errors. The standard diffusion training objective is

$$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$

and the preference objective preserves the DPO-style logistic comparison while operating at sampled denoising steps (Kang et al., 24 May 2025).

A key unifying feature of stable preference optimization is that many methods do not replace pairwise preference learning outright. Instead, they alter the surrounding training dynamics. Some are explicitly described as orthogonal add-ons that can be plugged into existing algorithms, as in the stable reference update and timestep-aware strategy for Diffusion-DPO, Diffusion-KTO, and DSPO (Kang et al., 24 May 2025). Others recast offline preference learning through Maximum Marginal Likelihood, bias–variance mixing, refreshable curricula, or bilevel optimization, but still retain the central objective of increasing likelihood assigned to preferred outputs relative to rejected ones (Najafi et al., 27 Oct 2025, Zhu et al., 6 Oct 2025, Jian et al., 10 Jul 2025).

2. Failure modes that motivate stability-aware preference optimization

Several papers argue that instability originates from structural properties of training rather than from the pairwise loss alone. In text-to-image diffusion, two specific issues are isolated: the reference model is usually frozen, which restricts exploration, and preference signals are not uniformly scaled across diffusion timesteps, which makes early-step preference learning weak. The same literature also identifies early denoising steps as semantically important but hard to learn from because implicit reward magnitudes are small there (Kang et al., 24 May 2025). In diffusion-video preference learning, standard Diffusion-DPO is further described as unstable because of a mismatch between the reverse and forward diffusion processes, high gradient variance in early noisy timesteps, and off-policy bias arising from the mismatch between optimization and data collection policies (2505.21893).

In language and reasoning models, instability is characterized differently but with related consequences. "Stable Preference Optimization for LLMs: A Bilevel Approach Beyond Direct Preference Optimization" argues that DPO is highly sensitive to initialization and can misallocate probability mass, improving the relative chosen-versus-rejected margin while still decreasing the absolute probability of the preferred output and shifting mass toward unrelated high-probability outputs inherited from SFT (Jian et al., 10 Jul 2025). "Not All Preferences Are Created Equal" argues that uniform treatment of preference pairs wastes computation on trivial pairs with negligible gradients and introduces noise from samples near uncertain decision boundaries, particularly in long-chain reasoning (Wu et al., 1 Feb 2026). "From Noisy Traces to Stable Gradients" identifies single-sampled reasoning traces as a major source of gradient variance because the statistically correct objective marginalizes over traces, but practical training usually optimizes one sampled trajectory (Zhu et al., 6 Oct 2025). "Preference as Reward, Maximum Preference Optimization with Importance Sampling" adds a different critique: DPO and IPO may fail to properly address the KL-regularization term because the support of the preference distribution is not equal to the reference distribution (Jiang et al., 2023).

A common misconception is that replacing RLHF with DPO is sufficient to obtain stable alignment. The literature is more specific: DPO is often described as more stable than RLHF, but multiple works still report collapse, overfitting, ineffective KL control, unstable probability dynamics, or high-variance gradients under realistic training conditions (Jiang et al., 2023, Jian et al., 10 Jul 2025, Zhu et al., 6 Oct 2025).

3. Diffusion-model formulations: bounded references, timestep awareness, and importance weighting

In text-to-image diffusion, stable preference optimization has been formulated as a modification of training dynamics rather than as a new standalone loss. "Rethinking Direct Preference Optimization in Diffusion Models" introduces a stable reference model update strategy that relaxes the frozen reference model by periodically updating it every $\tau$ training steps, while monitoring divergence from the initial pretrained model and freezing the reference near a threshold $\delta$ if that divergence becomes too large. Because exact KL over diffusion trajectories is intractable, the divergence is approximated through the forward noising process and implemented as the absolute value of an estimated difference in diffusion training losses. The same work adds a timestep-aware training strategy: timesteps are sampled from

$$t \sim \mathrm{Cat}\!\left(\frac{\gamma^t}{\sum_{t'} \gamma^{t'}}\right),$$

and the timestep weight is scheduled as

$$\lambda(t) = 1 + \alpha \cdot \mathrm{norm}\!\left(\frac{1}{\sqrt{\mathrm{SNR}(t)}}\right).$$

The stated purpose is to oversample earlier steps and increase emphasis where implicit reward scale is small, while keeping regularization bounded enough to avoid degeneration (Kang et al., 24 May 2025).

A related but distinct formulation appears in "SDPO: Importance-Sampled Direct Preference Optimization for Stable Diffusion Training." That paper first proposes DPO-CM, which clips and masks uninformative timesteps with

$$\tilde{w}(t)=\mathrm{clip}(w(t),1-\epsilon,1+\epsilon),$$

to suppress gradient flow from noisy regions where the forward and reverse paths diverge significantly. It then introduces SDPO, which uses importance sampling to correct off-policy bias and reweight learning toward informative steps. For diffusion trajectories, the paper uses a clipped inverse importance weight

$$\tilde{w}_\theta(x_t,t)= \mathrm{clip}\left(\frac{1}{w_\theta(x_t,t)},\,1-\epsilon,\,1+\epsilon\right),$$

and reports that mid timesteps, especially around $t\in[500,600]$, are the most stable and informative region of the trajectory. The paper explicitly distinguishes DPO-CM as a practical heuristic and SDPO as a principled importance-sampled framework (2505.21893).

Taken together, these diffusion works define stability through bounded exploration and timestep-aware credit assignment. One line keeps the reference adaptive but bounded; the other reweights the trajectory by informativeness and distribution correction. This suggests that, in diffusion alignment, stability is inseparable from denoising-time structure.

4. Language and reasoning models: bilevel coupling, dynamic selection, and bias–variance control

In LLM alignment, "Stable Preference Optimization for LLMs: A Bilevel Approach Beyond Direct Preference Optimization" introduces a method explicitly named Stable Preference Optimization (SPO). Its theoretical analysis derives first-order probability updates under DPO and argues that relative improvement does not imply absolute improvement for the preferred response. SPO addresses this by coupling preference optimization with supervised fine-tuning in a bilevel framework. The penalized objective combines $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$0, an SFT optimality penalty, and a regularizer

$$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$1

which is intended to discourage updates in which the rejected sample’s gradient norm dominates the preferred one. The method uses $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$2 for backbone weights and $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$3 for LoRA adapters, with $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$4-step lower-level SFT updates approximating $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$5 (Jian et al., 10 Jul 2025).

A different stabilization strategy is sample selection. SAGE, or Stability-Aware Gradient Efficiency, divides the dataset into easy, medium, and hard strata, builds refreshable candidate pools with a linear difficulty schedule, and then keeps only a high-SNR subset. Its fine-grained score is

$$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$6

where large gradient signal and low curvature are preferred, and length normalization prevents long responses from dominating. The retained batch is then defined by a top-$$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$7 filter on candidate scores (Wu et al., 1 Feb 2026).

For reasoning models with explicit chains of thought, "From Noisy Traces to Stable Gradients" argues that the statistically correct preference objective is over marginal answer probabilities, not sampled trace-answer trajectories. BVPO therefore mixes a high-variance trace-based estimator $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$8 with a low-variance empty-trace estimator $$\mathcal{L}_{\mathrm{DPO}} = -\, \mathbb{E}_{(x, y^w, y^l)} \log \sigma\!\big( r_\theta(x, y^w) - r_\theta(x, y^l) \big).$$9,

$$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$0

and proves that any nontrivial mixture strictly reduces trace-induced conditional variance, while also deriving a closed-form MSE-optimal $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$1 (Zhu et al., 6 Oct 2025).

Offline regularization and marginal-likelihood reformulations extend this picture. MPO proposes an off-policy objective with an offline regularizer over reference and pretraining data, motivated by the claim that DPO and IPO do not make KL regularization “truly effective” under support mismatch (Jiang et al., 2023). MMPO, by contrast, treats alignment as Maximum Marginal Likelihood estimation, uses log-sum-exp over pairwise approximations, and derives a weighted gradient in which the chosen response receives weight $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$2 and the rejected response receives weight $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$3. The paper attributes its greater stability to in-batch normalization and the log-sum-exp structure, and reports reduced sensitivity to $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$4 (Najafi et al., 27 Oct 2025).

5. Extensions to medical segmentation and combinatorial optimization

Stable preference optimization has also been generalized beyond text and diffusion generation.

Domain	Method	Stabilizing mechanism
Medical image segmentation	MAPO	Dropout-driven stochastic hypotheses, Dice-gap threshold $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$5, Dice + CE with DPO, online preference training
Neural combinatorial optimization	BOPO / POCO	Best-anchored preference pairs, objective-guided adaptive scaling, no reward model or reference policy

MAPO is described as a model-agnostic preference optimization framework for medical image segmentation. It generates multiple stochastic masks by activating dropout at inference time, selects a positive mask by maximum Dice overlap with ground truth, and selects a negative mask from a candidate set constrained by a Dice-gap threshold

$$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$6

Its total objective combines supervised segmentation losses with a DPO-style term,

$$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$7

and the paper explicitly states that DPO-only optimization can lead to unstable training dynamics, so the supervised term serves as an anchor. Online preference training regenerates preference data as the model improves (Nam et al., 17 Dec 2025).

BOPO, presented as Preference Optimization for Combinatorial Optimization, defines preference directly from objective values $$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$8, constructs best-anchored preference pairs from hybrid rollouts, and uses an objective-guided pairwise loss

$$\mathcal{L}_{\text{DDPM}} = \mathbb{E}_{x_0,\boldsymbol{\epsilon},t} \left[ \lambda(t)\left\lVert \boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta(x_t,t)\right\rVert_2^2 \right],$$9

The adaptive factor $\tau$0 scales gradients by objective difference, while length normalization prevents longer solutions from dominating. The method is explicitly described as reference-free and architecture-agnostic (Liao et al., 10 Mar 2025).

These extensions make the term “stable preference optimization” broader than LLM alignment alone. In segmentation, stability refers to pair construction and hybrid supervision; in combinatorial optimization, it refers to replacing sparse-return RL with pairwise learning grounded in exact objective values.

6. Empirical patterns, misconceptions, and unresolved questions

Across domains, the empirical pattern is that stability-oriented modifications often improve both optimization behavior and final alignment metrics. In text-to-image diffusion on SD1.5 / Pick-a-Pic v2, Diff-DPO average win rate improves from about 69.38 to 78.83, and preference accuracy of implicit rewards improves from near-random values to 0.6012 for Diff-DPO + Ours (Kang et al., 24 May 2025). In diffusion-video alignment on CogVideoX-2B, Diffusion-DPO reaches 81.16 at 500 steps but collapses to 67.28 at 1000 steps, whereas DPO-CM reaches 81.37 and 81.46, and SDPO reaches 81.53 and 81.68; SDPO also obtains 67% rank-1 in human evaluation (2505.21893). For reasoning models, SAGE improves Qwen2.5-7B-Instruct from 56.64 for DPO (Full) to 59.04, with smoother, lower-variance gradients, while BVPO reports gains of up to +7.8 points on AlpacaEval 2 and +6.8 points on Arena-Hard (Wu et al., 1 Feb 2026, Zhu et al., 6 Oct 2025).

Comparable effects are reported in LLM alignment and other structured prediction settings. SPO raises GSM8K accuracy on Qwen2.5-0.5B-Instruct from 44.3 for SFT and 35.4 for SFT + Vanilla DPO to 49.3 for SFT + SPO, and improves UltraFeedback summarization win rate on the 3B setting from 50.5 / 48.1 for SFT + Vanilla DPO to 55.3 / 51.0 for SFT + SPO (Jian et al., 10 Jul 2025). MMPO is reported as more stable with respect to $\tau$1 across models from 135M to 8B, with Llama3-8B reaching 66.48% by epoch 5 at $\tau$2 (Najafi et al., 27 Oct 2025). In medical segmentation, MAPO improves U-Net + MAPO on Kvasir from 65.06 → 74.08 Dice and improves UNETR + MAPO on Parse2022 from 60.01 → 62.76 Dice while reducing ASD from 18.15 → 15.47 (Nam et al., 17 Dec 2025). In combinatorial optimization, POCO reaches 0.37% gap on TSP100 versus 0.46% for POMO and 5.51% for SLIM in the non-augmented setting (Liao et al., 10 Mar 2025).

The literature also draws several boundaries around what stable preference optimization is not. It is not always a new preference loss: some methods are explicitly plug-in stabilization layers (Kang et al., 24 May 2025). It is not always reference-free: some methods strengthen or regularize the reference rather than removing it (Kang et al., 24 May 2025, 2505.21893). It is not always accompanied by a full stability theory: MPO is described as mostly derivational and heuristic rather than a formal convergence paper (Jiang et al., 2023), and SPO states that its theory is mainly single-step and that broader scalability remains to be further validated (Jian et al., 10 Jul 2025). The resulting picture is technically heterogeneous but conceptually consistent: stable preference optimization treats alignment stability as a property of the entire optimization pipeline, including reference evolution, data support, trajectory structure, estimator variance, and supervision design.