Anchored Preference Optimization Methods

• Anchored Preference Optimization (APO) is a set of methods that compute preference rewards relative to a fixed or periodically updated reference, ensuring controlled and stable policy updates.
• APO employs techniques like anchored log-ratio scoring, dual KL penalties, and groupwise invariance across variants such as APO-zero, BAPO, and ADPO to balance adaptation with retention.
• Empirical studies show APO improves performance in language model alignment by up to 7.65% and has been successfully adapted for applications in video reasoning, image super-resolution, and audio processing.

Searching arXiv for papers on Anchored Preference Optimization and related anchored preference optimization methods. Anchored Preference Optimization (APO) denotes a class of preference-optimization methods that introduce an explicit anchor into the alignment objective so that optimization is carried out relative to a fixed or periodically updated reference, rather than in unconstrained policy space. In the language-model setting, the anchor is typically a reference policy $\pi_{\mathrm{ref}}$, a base policy $\pi_{\mathrm{base}}$, or both; in other modalities, the anchor can instead be a low-resolution input, an inference-time timestep schedule, a short clip within a long video, or a deep acoustic feature representation (Lee et al., 2024, D'Oosterlinck et al., 2024, Zixian, 21 Oct 2025, Fei et al., 6 Mar 2026, Huang et al., 2 Feb 2026, Peng et al., 11 Jul 2026). Across these formulations, anchoring is introduced to mitigate catastrophic forgetting, stabilize optimization, impose an implicit trust region, or prevent reward hacking. The term is therefore best understood as a family resemblance across methods rather than a single universally standardized objective.

1. Definition and scope

In one influential formulation, APO is introduced as a “controllable and more stable alignment objective” for preference datasets of triples $(x,y_l,y_w)$, where $y_w$ is the preferred response and $y_l$ the dispreferred response (D'Oosterlinck et al., 2024). The model likelihood is written as $\pi_\theta(y\mid x)$, and the reference model as $\pi_{\mathrm{ref}}(y\mid x)$. The reward is defined by the anchored log-ratio

$$r_\theta(x,y) = \beta \cdot \log \left[\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\right],$$

with $\beta>0$ a scaling parameter (D'Oosterlinck et al., 2024). In this usage, the anchor distribution $\pi_{\mathrm{ref}}(y\mid x)$ is the “before-training” model, and anchoring to $\pi_{\mathrm{base}}$0 “prevents runaway increases in log-likelihood and gives us a zero-point for rewards” (D'Oosterlinck et al., 2024).

A second line of work uses anchoring to combat forgetting during personalization. “BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in LLMs Personalization” introduces two anchors in the policy-optimization objective: a reference anchor $\pi_{\mathrm{base}}$1, “typically the most recent version of the model before the current round of preference updates,” and a base anchor $\pi_{\mathrm{base}}$2, “a frozen copy of the original pretrained (or pre-preference-tuned) model” (Lee et al., 2024). Intuitively, $\pi_{\mathrm{base}}$3 guides the model to make only incremental adjustments for the current user, while $\pi_{\mathrm{base}}$4 ensures the model does not stray too far from its foundational knowledge (Lee et al., 2024).

A later generalization, Anchored Direct Preference Optimization (ADPO), describes anchoring as a unified framework that “generalizes Direct Preference Optimization (DPO) with soft preferences, reference-policy anchoring, and groupwise extensions” (Zixian, 21 Oct 2025). In that setting, the anchored student distribution is constructed from relative logits $\pi_{\mathrm{base}}$5, and the resulting objectives are groupwise-shift-invariant and implicitly regularized (Zixian, 21 Oct 2025).

This suggests that APO is most coherently defined by its geometry: optimization is expressed in anchored coordinates, usually through log-probability differences or KL penalties to a trusted reference.

2. Core mathematical forms

The simplest anchored reward parameterization used in language-model alignment is

$\pi_{\mathrm{base}}$6

On top of this reward, “Anchored Preference Optimization and Contrastive Revisions” defines two primary APO variants (D'Oosterlinck et al., 2024):

$\pi_{\mathrm{base}}$7

and

$\pi_{\mathrm{base}}$8

where $\pi_{\mathrm{base}}$9 is the logistic (D'Oosterlinck et al., 2024). Minimizing these losses ensures that APO-zero increases $(x,y_l,y_w)$0 and decreases $(x,y_l,y_w)$1, whereas APO-down decreases both $(x,y_l,y_w)$2 and $(x,y_l,y_w)$3, with the latter decreasing more (D'Oosterlinck et al., 2024).

In BAPO, the anchor enters through dual KL penalties. The expected preference reward is combined with KL constraints to both the reference model and the base model:

$(x,y_l,y_w)$4

where $(x,y_l,y_w)$5 is the scalar preference reward and $(x,y_l,y_w)$6 balance adaptation versus retention (Lee et al., 2024). Each KL term is defined as

$(x,y_l,y_w)$7

The PPO-style surrogate is then augmented by the same dual-KL structure (Lee et al., 2024).

ADPO moves the anchor directly into DPO-style preference losses. For soft pairwise preferences, the anchored Soft-DPO loss is

$(x,y_l,y_w)$8

where $(x,y_l,y_w)$9 and $y_w$0 (Zixian, 21 Oct 2025). For listwise learning, the anchored student distribution is

$y_w$1

and the listwise cross-entropy is

$y_w$2

Because $y_w$3 is invariant to adding the same constant to all logits, the loss is groupwise-shift-invariant (Zixian, 21 Oct 2025).

A plausible implication is that the principal mathematical distinction among APO variants is not whether they use contrastive, pairwise, listwise, PPO-style, or DPO-style learning, but where the optimization is centered.

3. Stabilization, control, and retention

A recurrent justification for anchoring is that standard preference optimization is underspecified or unstable when absolute policy movement is not controlled. In the controllable APO objective of (D'Oosterlinck et al., 2024), DPO is described as enforcing only that $y_w$4 should increase, “leaving absolute movements under-determined.” APO instead “explicitly specifies direction for $y_w$5 and $y_w$6,” and the logistic nonlinearity makes gradients saturate when $y_w$7, yielding “stable, self-limiting updates” (D'Oosterlinck et al., 2024). The derivative factor $y_w$8 peaks at $y_w$9 and tends to $y_l$0 as $y_l$1, which implements a “natural curriculum” in which gradients are largest for uncertain examples (D'Oosterlinck et al., 2024).

In BAPO, the main stabilization target is catastrophic forgetting under personalized alignment. The paper states that previous approaches using only the KL constraint between the reference model and the policy model “fail to maintain general knowledge and alignment when facing personalized preferences” (Lee et al., 2024). BAPO’s dual constraints penalize deviation both from the immediate past and from the original base, thereby guarding against cumulative drift (Lee et al., 2024). The base anchor never changes, whereas the reference anchor is periodically “snapped” to the current policy at the end of each iteration (Lee et al., 2024).

ADPO gives a more formal account of why anchoring stabilizes optimization. A second-order Taylor expansion of the listwise cross-entropy yields

$y_l$2

under small deviations $y_l$3 (Zixian, 21 Oct 2025). The paper therefore concludes that anchoring acts like a trust region penalizing $y_l$4 under $y_l$5 (Zixian, 21 Oct 2025).

The same theme appears in diffusion alignment. “Rethinking Direct Preference Optimization in Diffusion Models” describes a “stable reference model update strategy that relaxes the frozen reference model, encouraging exploration while maintaining a stable optimization anchor through reference model regularization” (Kang et al., 24 May 2025). There, the anchor is allowed to move only when an estimated divergence from the original pretrained model remains below a threshold $y_l$6 (Kang et al., 24 May 2025).

4. Variants and generalizations

Anchored methods have been extended along several axes.

ADPO generalizes standard DPO with “soft preference probabilities that encode uncertainty and mitigate gradient drift,” “arbitrary reference-policy anchors that stabilize training via groupwise shift invariance and implicit KL regularization,” and “listwise preference modeling through Plackett-Luce distributions” (Zixian, 21 Oct 2025). The paper proves that standard DPO, Bradley–Terry objectives, and Top-1-vs-Rest formulations emerge as special cases (Zixian, 21 Oct 2025). It also gives three concrete variants: pairwise anchored Soft-DPO, listwise anchored Soft-DPO with raw rewards, and listwise anchored Soft-DPO with KDE smoothing for heavy-tailed outliers (Zixian, 21 Oct 2025).

“APO: Alpha-Divergence Preference Optimization” extends anchored optimization from KL-type objectives to a continuous family of Csiszár–Amari $y_l$7-divergences (Zixian, 28 Dec 2025). In that formulation, one works in anchored coordinates with

$y_l$8

and minimizes

$y_l$9

As $\pi_\theta(y\mid x)$0, this recovers $\pi_\theta(y\mid x)$1, and as $\pi_\theta(y\mid x)$2, it recovers $\pi_\theta(y\mid x)$3 (Zixian, 28 Dec 2025). The method introduces a reward-and-confidence-guarded $\pi_\theta(y\mid x)$4 schedule so that the divergence regime changes only when the policy is both confident and improving (Zixian, 28 Dec 2025).

The acronym APO is also used for conceptually distinct frameworks. “Accelerated Preference Optimization for LLM Alignment” uses Nesterov’s momentum within iterative preference optimization and defines an extrapolation step in policy space (He et al., 2024). “Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game” instead casts alignment as an RM-LLM min-max game with alternating updates of the reward model and the LLM (Cheng et al., 2023). These methods are not anchor-centered in the same sense as the anchored-objective family, even though they share the acronym.

This suggests that disambiguation by full title rather than acronym is necessary in the APO literature.

5. Empirical findings in language-model alignment

For the controllable APO objective of (D'Oosterlinck et al., 2024), experiments use Llama-3-8B-Instruct, multiple 32K preference datasets sampled from UltraFeedback, 18 epochs of training, and MixEval-Hard evaluation (D'Oosterlinck et al., 2024). The paper reports that “CLAIR + APO-zero yields the largest gain (+7.65 pp), closing 45 % of the gap to GPT-4-turbo with only 32 K pairs,” and that across all datasets except the intentionally noisy Stronger Preferred, some APO variant is best (D'Oosterlinck et al., 2024). The best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, while the baseline Llama-3-8B-Instruct score is 41.45% overall on MixEval-Hard and the gap to GPT-4-turbo is 17% (D'Oosterlinck et al., 2024).

BAPO reports that the extent of knowledge loss under personalized preference optimization “varies significantly with preference heterogeneity” and that BAPO “effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment” (Lee et al., 2024). The focused summary states experimental findings of “~25% less drop in zero-shot performance on held-out tasks compared to single-anchor PPO,” “higher preference-alignment reward on both in-domain and out-of-domain prompts,” and that removing the base anchor immediately recovers standard drift issues (Lee et al., 2024).

ADPO reports improvements in contextual bandits and sequential reinforcement learning. In contextual bandits, anchoring improves WinMass by 38–63% over standard DPO, while KDE smoothing achieves 0.68 versus 0.32 under heavy-tailed contamination (Zixian, 21 Oct 2025). In sequential reinforcement learning, anchoring improves noisy-preference performance by 15–29%, and practical guidance recommends pairwise anchored Soft-DPO for clean or moderate noise and KDE-based listwise ADPO for extreme contamination (Zixian, 21 Oct 2025).

Alpha-Divergence Preference Optimization evaluates Qwen3-1.7B on math-level3 problems from the MATH dataset. The paper states that all methods achieve similar final mean reward, around 0.65, without catastrophic collapse, and that the APO variants match GSPO and ADPO-Softmax in peak performance while exhibiting smooth, stable learning (Zixian, 28 Dec 2025).

6. Cross-modal instantiations

Anchoring has also been adapted to several non-text modalities.

In generative real-world image super-resolution, “LucidNFT” treats the degraded low-resolution image $\pi_\theta(y\mid x)$5 as the anchor and introduces LR-anchored faithfulness through LucidConsistency, defined by

$\pi_\theta(y\mid x)$6

This score becomes one component of a multi-dimensional reward, and objective-wise group normalization is used before fusion to prevent “advantage collapse” (Fei et al., 6 Mar 2026).

In long-form video reasoning, “LongVPO” instantiates APO in two stages. In Stage 1, questions are anchored to an individual short clip and preference triples are created by contrasting answers generated from the anchor clip and distractor clips; the reference model’s scoring over long contexts is approximated by evaluating only the anchor clip (Huang et al., 2 Feb 2026). The paper states that this “reduces memory/computation roughly by a factor of $\pi_\theta(y\mid x)$7,” yielding approximately 70% faster DPO training in practice (Huang et al., 2 Feb 2026). Stage 2 then aligns the model on multi-segment reasoning tasks over long videos using DPO with a reference frozen at the end of Stage 1 (Huang et al., 2 Feb 2026).

In text-to-image diffusion, APO uses a stable reference-model update and timestep-aware training. A fixed reference model is relaxed by periodically replacing $\pi_\theta(y\mid x)$8 with the current $\pi_\theta(y\mid x)$9 if an estimated divergence from the original pretrained model remains below threshold $\pi_{\mathrm{ref}}(y\mid x)$0 (Kang et al., 24 May 2025). The same work adds oversampling and reward-scale scheduling over timesteps to compensate for early-step magnitude imbalance (Kang et al., 24 May 2025).

In video diffusion transformers, “Diffusion-APO” anchors training-time timestep sampling to the actual inference trajectory and combines a DPO loss with an anchoring regularizer

$\pi_{\mathrm{ref}}(y\mid x)$1

approximated in practice by an $\pi_{\mathrm{ref}}(y\mid x)$2 penalty on predicted noise differences (Zhu et al., 8 May 2026). The framework also introduces half-online anchoring, where each stage freezes $\pi_{\mathrm{ref}}(y\mid x)$3 to the current model and keeps it fixed throughout that stage (Zhu et al., 8 May 2026).

In streaming target speaker extraction, APO is realized by WavLM-anchored DPO. Preference pairs are constructed by ranking candidate extractions with WavLM cosine similarity to a clean target-speaker reference, rather than by DNSMOS or WER (Peng et al., 11 Jul 2026). The paper states that deep acoustic feature anchoring resists reward hacking and, under a 560 ms streaming chunk size, yields a 10.9% relative intelligibility improvement, reducing word error rate from 0.138 to 0.123 while also improving audio quality and speaker similarity (Peng et al., 11 Jul 2026).

7. Limitations, misconceptions, and open directions

A common misconception is that anchoring is synonymous with a single frozen reference policy. The literature is more varied. BAPO uses both a moving reference anchor and a frozen base anchor (Lee et al., 2024). Diffusion APO periodically updates the reference under a safe-update rule (Kang et al., 24 May 2025). LongVPO approximates reference scoring by evaluating only the anchor clip rather than the full long context (Huang et al., 2 Feb 2026). LucidNFT treats the low-resolution input itself as the anchor for faithfulness (Fei et al., 6 Mar 2026).

A second misconception is that all APO methods are DPO variants. Some are not. BAPO is PPO-style with dual KL penalties (Lee et al., 2024). The controllable APO objective of (D'Oosterlinck et al., 2024) defines APO-zero and APO-down directly in terms of anchored rewards rather than the standard DPO log-sigmoid margin. Diffusion-APO augments pairwise DPO with an explicit anchoring regularizer over reverse kernels (Zhu et al., 8 May 2026). Alpha-Divergence Preference Optimization replaces a single KL geometry with a continuum of anchored $\pi_{\mathrm{ref}}(y\mid x)$4-divergences (Zixian, 28 Dec 2025).

Reported limitations also differ by formulation. BAPO notes that storing the full base model can increase memory cost, and that a fixed base anchor may be too conservative if the foundational model becomes outdated (Lee et al., 2024). The controllable APO work emphasizes that APO-zero and APO-down should be selected according to whether the preferred response is above or below the current model, to avoid overshooting or undershooting (D'Oosterlinck et al., 2024). Alpha-Divergence Preference Optimization introduces extra hyperparameters such as $\pi_{\mathrm{ref}}(y\mid x)$5, $\pi_{\mathrm{ref}}(y\mid x)$6, $\pi_{\mathrm{ref}}(y\mid x)$7, $\pi_{\mathrm{ref}}(y\mid x)$8, and $\pi_{\mathrm{ref}}(y\mid x)$9 (Zixian, 28 Dec 2025). Diffusion-APO notes dependence on the fidelity of preference signals and increased computational pipeline complexity (Zhu et al., 8 May 2026).

Future directions explicitly named in the literature include adaptive anchor weighting schedules, hierarchical anchoring with multiple intermediate snapshots, meta-learning the anchor weights in BAPO (Lee et al., 2024), empirical extension of accelerated preference optimization to SPPO and adaptive momentum schedules (He et al., 2024), and larger-model, richer-task studies for alpha-divergence APO (Zixian, 28 Dec 2025). A plausible implication is that the most active frontier is no longer whether to anchor, but how to choose the anchor, when to move it, and in which coordinate system the preference projection should be performed.