---
title: Anchored Preference Optimization Methods
url: https://www.emergentmind.com/topics/anchored-preference-optimization-apo
type: topic
---

# Anchored Preference Optimization Methods

Searching arXiv for recent papers on Anchored Preference Optimization and related anchored preference optimization methods.
Anchored Preference Optimization (APO) denotes a class of preference-optimization methods that introduce an explicit anchor into the alignment objective so that optimization is carried out relative to a fixed or periodically updated reference, rather than in unconstrained policy space. In the language-model setting, the anchor is typically a reference policy $\pi_{\mathrm{ref}}$, a base policy $\pi_{\mathrm{base}}$, or both; in other modalities, the anchor can instead be a low-resolution input, an inference-time timestep schedule, a short clip within a long video, or a deep acoustic feature representation [2407.00693], [2408.06266], [2510.18913], [2603.05947], [2602.02341], [2607.10191]. Across these formulations, anchoring is introduced to mitigate catastrophic forgetting, stabilize optimization, impose an implicit trust region, or prevent reward hacking. The term is therefore best understood as a family resemblance across methods rather than a single universally standardized objective.

## 1. Definition and scope

In one influential formulation, APO is introduced as a “controllable and more stable alignment objective” for preference datasets of triples $(x,y_l,y_w)$, where $y_w$ is the preferred response and $y_l$ the dispreferred response [2408.06266]. The model likelihood is written as $\pi_\theta(y\mid x)$, and the reference model as $\pi_{\mathrm{ref}}(y\mid x)$. The reward is defined by the anchored log-ratio
$$
r_\theta(x,y) = \beta \cdot \log \left[\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\right],
$$
with $\beta>0$ a scaling parameter [2408.06266]. In this usage, the anchor distribution $\pi_{\mathrm{ref}}(y\mid x)$ is the “before-training” model, and anchoring to $\pi_{\mathrm{ref}}$ “prevents runaway increases in log-likelihood and gives us a zero-point for rewards” [2408.06266].

A second line of work uses anchoring to combat forgetting during personalization. “BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization” introduces two anchors in the policy-optimization objective: a reference anchor $\pi_{\mathrm{ref}}$, “typically the most recent version of the model before the current round of preference updates,” and a base anchor $\pi_{\mathrm{base}}$, “a frozen copy of the original pretrained (or pre-preference-tuned) model” [2407.00693]. Intuitively, $\pi_{\mathrm{ref}}$ guides the model to make only incremental adjustments for the current user, while $\pi_{\mathrm{base}}$ ensures the model does not stray too far from its foundational knowledge [2407.00693].

A later generalization, Anchored Direct Preference Optimization (ADPO), describes anchoring as a unified framework that “generalizes Direct Preference Optimization (DPO) with soft preferences, reference-policy anchoring, and groupwise extensions” [2510.18913]. In that setting, the anchored student distribution is constructed from relative logits $s_i-s_i^{\mathrm{ref}}$, and the resulting objectives are groupwise-shift-invariant and implicitly regularized [2510.18913].

This suggests that APO is most coherently defined by its geometry: optimization is expressed in anchored coordinates, usually through log-probability differences or KL penalties to a trusted reference.

## 2. Core mathematical forms

The simplest anchored reward parameterization used in language-model alignment is
$$
r_\theta(x,y) = \beta \cdot \log \left[\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\right].
$$
On top of this reward, “Anchored Preference Optimization and Contrastive Revisions” defines two primary APO variants [2408.06266]:
$$
\mathcal{L}_{\text{APO-zero}(x,y_w,y_l;\,\theta)}
=
-\,\sigma\bigl(r_\theta(x,y_w)\bigr)
\;+\;\sigma\bigl(r_\theta(x,y_l)\bigr),
$$
and
$$
\mathcal{L}_{\text{APO-down}(x,y_w,y_l;\,\theta)}
=
\phantom{-}\,\sigma\bigl(r_\theta(x,y_w)\bigr)
\;-\;\sigma\bigl(r_\theta(x,y_w)\;-\;r_\theta(x,y_l)\bigr),
$$
where $\sigma(\cdot)=1/(1+e^{-x})$ is the logistic [2408.06266]. Minimizing these losses ensures that APO-zero increases $\pi_\theta(y_w\mid x)$ and decreases $\pi_\theta(y_l\mid x)$, whereas APO-down decreases both $\pi_\theta(y_w\mid x)$ and $\pi_\theta(y_l\mid x)$, with the latter decreasing more [2408.06266].

In BAPO, the anchor enters through dual KL penalties. The expected preference reward is combined with KL constraints to both the reference model and the base model:
$$
L(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\bigl[r(\tau)\bigr]
-\alpha\,D_{\mathrm{KL}\bigl(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\bigr)}
-\beta\,D_{\mathrm{KL}\bigl(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{base}}(\cdot\mid x)\bigr)},
$$
where $r(\tau)$ is the scalar preference reward and $\alpha,\beta>0$ balance adaptation versus retention [2407.00693]. Each KL term is defined as
$$
D_{\mathrm{KL}}(\pi_1\Vert\pi_2)=\sum_a \pi_1(a\mid x)\log\frac{\pi_1(a\mid x)}{\pi_2(a\mid x)}.
$$
The PPO-style surrogate is then augmented by the same dual-KL structure [2407.00693].

ADPO moves the anchor directly into DPO-style preference losses. For soft pairwise preferences, the anchored Soft-DPO loss is
$$
\ell_{ij}^{\;Soft\text{-}DPO}
=
\log(1 + e^{\beta\,( \Delta_\theta^{ij}-\Delta_{\mathrm{ref}}^{ij})})
- q_{ij}\;\beta\,(\Delta_\theta^{ij}-\Delta_{\mathrm{ref}}^{ij}),
$$
where $\Delta_\theta^{ij}=s_i-s_j$ and $\Delta_{\mathrm{ref}}^{ij}=s_i^{\mathrm{ref}}-s_j^{\mathrm{ref}}$ [2510.18913]. For listwise learning, the anchored student distribution is
$$
\tilde p_\theta(i\mid S_x)=
\frac{\exp((s_i-s_i^{\mathrm{ref}})/\tau)}
{\sum_{j\in S_x}\exp((s_j-s_j^{\mathrm{ref}})/\tau)},
$$
and the listwise cross-entropy is
$$
L^{\mathrm{ref}}_{\mathrm{group}}
=
-E_{x,S_x}\Bigl[\sum_{i\in S_x} q(i\mid S_x)\,\log \tilde p_\theta(i\mid S_x)\Bigr].
$$
Because $\Delta_\theta-\Delta_{\mathrm{ref}}$ is invariant to adding the same constant to all logits, the loss is groupwise-shift-invariant [2510.18913].

A plausible implication is that the principal mathematical distinction among APO variants is not whether they use contrastive, pairwise, listwise, PPO-style, or DPO-style learning, but where the optimization is centered.

## 3. Stabilization, control, and retention

A recurrent justification for anchoring is that standard preference optimization is underspecified or unstable when absolute policy movement is not controlled. In the controllable APO objective of [2408.06266], DPO is described as enforcing only that $r_\theta(y_w)-r_\theta(y_l)$ should increase, “leaving absolute movements under-determined.” APO instead “explicitly specifies direction for $y_w$ and $y_l$,” and the logistic nonlinearity makes gradients saturate when $|r|\gg 0$, yielding “stable, self-limiting updates” [2408.06266]. The derivative factor $\delta(r)=\sigma(r)(1-\sigma(r))$ peaks at $r=0$ and tends to $0$ as $|r|\to\infty$, which implements a “natural curriculum” in which gradients are largest for uncertain examples [2408.06266].

In BAPO, the main stabilization target is catastrophic forgetting under personalized alignment. The paper states that previous approaches using only the KL constraint between the reference model and the policy model “fail to maintain general knowledge and alignment when facing personalized preferences” [2407.00693]. BAPO’s dual constraints penalize deviation both from the immediate past and from the original base, thereby guarding against cumulative drift [2407.00693]. The base anchor never changes, whereas the reference anchor is periodically “snapped” to the current policy at the end of each iteration [2407.00693].

ADPO gives a more formal account of why anchoring stabilizes optimization. A second-order Taylor expansion of the listwise cross-entropy yields
$$
KL[q\|\tilde p_\theta]
\approx \frac{1}{2}\frac{1}{\tau^2}E_q[\delta_i^2] + o(\|\delta\|^2),
$$
under small deviations $\delta_i=(s_i-s_i^{\mathrm{ref}})-E_q[s-s^{\mathrm{ref}}]$ [2510.18913]. The paper therefore concludes that anchoring acts like a trust region penalizing $\|s-s^{\mathrm{ref}}\|^2$ under $q$ [2510.18913].

The same theme appears in diffusion alignment. “Rethinking Direct Preference Optimization in Diffusion Models” describes a “stable reference model update strategy that relaxes the frozen reference model, encouraging exploration while maintaining a stable optimization anchor through reference model regularization” [2505.18736]. There, the anchor is allowed to move only when an estimated divergence from the original pretrained model remains below a threshold $\delta$ [2505.18736].

## 4. Variants and generalizations

Anchored methods have been extended along several axes.

ADPO generalizes standard DPO with “soft preference probabilities that encode uncertainty and mitigate gradient drift,” “arbitrary reference-policy anchors that stabilize training via groupwise shift invariance and implicit KL regularization,” and “listwise preference modeling through Plackett-Luce distributions” [2510.18913]. The paper proves that standard DPO, Bradley–Terry objectives, and Top-1-vs-Rest formulations emerge as special cases [2510.18913]. It also gives three concrete variants: pairwise anchored Soft-DPO, listwise anchored Soft-DPO with raw rewards, and listwise anchored Soft-DPO with KDE smoothing for heavy-tailed outliers [2510.18913].

“APO: Alpha-Divergence Preference Optimization” extends anchored optimization from KL-type objectives to a continuous family of Csiszár–Amari $\alpha$-divergences [2512.22953]. In that formulation, one works in anchored coordinates with
$$
u_i = \frac{\ell_i-\ell_i^{\mathrm{ref}}}{\tau},
\qquad
p_\theta(i\mid S_x)=\mathrm{softmax}(u)_i,
$$
and minimizes
$$
D_\alpha(q\|p)
=
\frac{1}{\alpha(1-\alpha)}
\left[
1-\sum_{i\in S} q(i)^\alpha p(i)^{1-\alpha}
\right].
$$
As $\alpha\to 1$, this recovers $KL(q\|p)$, and as $\alpha\to 0$, it recovers $KL(p\|q)$ [2512.22953]. The method introduces a reward-and-confidence-guarded $\alpha$ schedule so that the divergence regime changes only when the policy is both confident and improving [2512.22953].

The acronym APO is also used for conceptually distinct frameworks. “Accelerated Preference Optimization for Large Language Model Alignment” uses Nesterov’s momentum within iterative preference optimization and defines an extrapolation step in policy space [2410.06293]. “Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game” instead casts alignment as an RM-LLM min-max game with alternating updates of the reward model and the language model [2311.08045]. These methods are not anchor-centered in the same sense as the anchored-objective family, even though they share the acronym.

This suggests that disambiguation by full title rather than acronym is necessary in the APO literature.

## 5. Empirical findings in language-model alignment

For the controllable APO objective of [2408.06266], experiments use Llama-3-8B-Instruct, multiple 32K preference datasets sampled from UltraFeedback, 18 epochs of training, and MixEval-Hard evaluation [2408.06266]. The paper reports that “CLAIR + APO-zero yields the largest gain (+7.65 pp), closing 45 % of the gap to GPT-4-turbo with only 32 K pairs,” and that across all datasets except the intentionally noisy Stronger Preferred, some APO variant is best [2408.06266]. The best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, while the baseline Llama-3-8B-Instruct score is 41.45% overall on MixEval-Hard and the gap to GPT-4-turbo is 17% [2408.06266].

BAPO reports that the extent of knowledge loss under personalized preference optimization “varies significantly with preference heterogeneity” and that BAPO “effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment” [2407.00693]. The focused summary states experimental findings of “~25% less drop in zero-shot performance on held-out tasks compared to single-anchor PPO,” “higher preference-alignment reward on both in-domain and out-of-domain prompts,” and that removing the base anchor immediately recovers standard drift issues [2407.00693].

ADPO reports improvements in contextual bandits and sequential reinforcement learning. In contextual bandits, anchoring improves WinMass by 38–63% over standard DPO, while KDE smoothing achieves 0.68 versus 0.32 under heavy-tailed contamination [2510.18913]. In sequential reinforcement learning, anchoring improves noisy-preference performance by 15–29%, and practical guidance recommends pairwise anchored Soft-DPO for clean or moderate noise and KDE-based listwise ADPO for extreme contamination [2510.18913].

Alpha-Divergence Preference Optimization evaluates Qwen3-1.7B on math-level3 problems from the MATH dataset. The paper states that all methods achieve similar final mean reward, around 0.65, without catastrophic collapse, and that the APO variants match GSPO and ADPO-Softmax in peak performance while exhibiting smooth, stable learning [2512.22953].

## 6. Cross-modal instantiations

Anchoring has also been adapted to several non-text modalities.

In generative real-world image super-resolution, “LucidNFT” treats the degraded low-resolution image $c=x_{lr}$ as the anchor and introduces LR-anchored faithfulness through LucidConsistency, defined by
$$
\mathcal C(x_{lr},x_{sr})
=
\cos\!\bigl(g(E(x_{lr})),\,g(E(x_{sr}))\bigr)\in[-1,1].
$$
This score becomes one component of a multi-dimensional reward, and objective-wise group normalization is used before fusion to prevent “advantage collapse” [2603.05947].

In long-form video reasoning, “LongVPO” instantiates APO in two stages. In Stage 1, questions are anchored to an individual short clip and preference triples are created by contrasting answers generated from the anchor clip and distractor clips; the reference model’s scoring over long contexts is approximated by evaluating only the anchor clip [2602.02341]. The paper states that this “reduces memory/computation roughly by a factor of $k$,” yielding approximately 70% faster DPO training in practice [2602.02341]. Stage 2 then aligns the model on multi-segment reasoning tasks over long videos using DPO with a reference frozen at the end of Stage 1 [2602.02341].

In text-to-image diffusion, APO uses a stable reference-model update and timestep-aware training. A fixed reference model is relaxed by periodically replacing $\theta_{\mathrm{ref}}$ with the current $\theta$ if an estimated divergence from the original pretrained model remains below threshold $\delta$ [2505.18736]. The same work adds oversampling and reward-scale scheduling over timesteps to compensate for early-step magnitude imbalance [2505.18736].

In video diffusion transformers, “Diffusion-APO” anchors training-time timestep sampling to the actual inference trajectory and combines a DPO loss with an anchoring regularizer
$$
\Omega_{\mathrm{anchor}}(\theta)
=
E_{x_t,t,c}\bigl[
D_{\mathrm{KL}}(p_\theta(x_{t-1}\mid x_t,c)\|p_{\mathrm{ref}}(x_{t-1}\mid x_t,c))
\bigr],
$$
approximated in practice by an $\ell_2$ penalty on predicted noise differences [2605.07503]. The framework also introduces half-online anchoring, where each stage freezes $\pi_{\mathrm{ref}}$ to the current model and keeps it fixed throughout that stage [2605.07503].

In streaming target speaker extraction, APO is realized by WavLM-anchored DPO. Preference pairs are constructed by ranking candidate extractions with WavLM cosine similarity to a clean target-speaker reference, rather than by DNSMOS or WER [2607.10191]. The paper states that deep acoustic feature anchoring resists reward hacking and, under a 560 ms streaming chunk size, yields a 10.9% relative intelligibility improvement, reducing word error rate from 0.138 to 0.123 while also improving audio quality and speaker similarity [2607.10191].

## 7. Limitations, misconceptions, and open directions

A common misconception is that anchoring is synonymous with a single frozen reference policy. The literature is more varied. BAPO uses both a moving reference anchor and a frozen base anchor [2407.00693]. Diffusion APO periodically updates the reference under a safe-update rule [2505.18736]. LongVPO approximates reference scoring by evaluating only the anchor clip rather than the full long context [2602.02341]. LucidNFT treats the low-resolution input itself as the anchor for faithfulness [2603.05947].

A second misconception is that all APO methods are DPO variants. Some are not. BAPO is PPO-style with dual KL penalties [2407.00693]. The controllable APO objective of [2408.06266] defines APO-zero and APO-down directly in terms of anchored rewards rather than the standard DPO log-sigmoid margin. Diffusion-APO augments pairwise DPO with an explicit anchoring regularizer over reverse kernels [2605.07503]. Alpha-Divergence Preference Optimization replaces a single KL geometry with a continuum of anchored $\alpha$-divergences [2512.22953].

Reported limitations also differ by formulation. BAPO notes that storing the full base model can increase memory cost, and that a fixed base anchor may be too conservative if the foundational model becomes outdated [2407.00693]. The controllable APO work emphasizes that APO-zero and APO-down should be selected according to whether the preferred response is above or below the current model, to avoid overshooting or undershooting [2408.06266]. Alpha-Divergence Preference Optimization introduces extra hyperparameters such as $\alpha_{\min}$, $\alpha_{\max}$, $\rho$, $\lambda$, and $s_R$ [2512.22953]. Diffusion-APO notes dependence on the fidelity of preference signals and increased computational pipeline complexity [2605.07503].

Future directions explicitly named in the literature include adaptive anchor weighting schedules, hierarchical anchoring with multiple intermediate snapshots, meta-learning the anchor weights in BAPO [2407.00693], empirical extension of accelerated preference optimization to SPPO and adaptive momentum schedules [2410.06293], and larger-model, richer-task studies for alpha-divergence APO [2512.22953]. A plausible implication is that the most active frontier is no longer whether to anchor, but how to choose the anchor, when to move it, and in which coordinate system the preference projection should be performed.

Source: https://www.emergentmind.com/topics/anchored-preference-optimization-apo