Distributionally Robust Listwise Preference Optimization

Abstract: Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from $K!$ enumeration to $O(K\log K)$. This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global $ε$-suboptimality with $O(ε^{-2})$ sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and $\widetilde O(ε^{-2})$ Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.

• The paper introduces a robust listwise preference optimization model that mitigates ranking-label uncertainty using a total-variation ball framework.
• It leverages the Plackett–Luce model to efficiently compute adversarial worst-case rankings with O(K log K) complexity, enhancing computational tractability.
• Empirical results demonstrate that the method improves ranking correlation and stabilizes performance under noisy annotations in both offline and online settings.

Distributionally Robust Listwise Preference Optimization: Technical Summary

Introduction and Motivation

The paper "Distributionally Robust Listwise Preference Optimization" (2607.01715) addresses robustness in listwise preference optimization for LLM alignment, focusing specifically on ranking-label uncertainty. Unlike prior robust preference optimization approaches that primarily target pairwise comparisons and perturbations at the data, prompt, or pairwise preference level, this work considers the ambiguity inherent in observed rankings over a candidate response list. Such ambiguities arise from annotator inconsistencies, near-ties, incomplete rankwise feedback, or noise in reward models. The proposed method aims for robustness at the conditional ranking-label level given a fixed candidate set per prompt.

Methodology

Plackett–Luce Listwise Robustification

The foundation is the Plackett–Luce (PL) model, a strict generalization of the Bradley–Terry (BT) model, designed to capture listwise ranking information for $K \geq 2$ candidates. The listwise loss aggregates sequence-level ordering information, thus leveraging richer supervision than pairwise methods.

The proposed robustification defines a total-variation (TV) ball around the empirical ranking label, conditional on the realized candidate list. For a prompt $x$ and candidate list $Y = (y_1, ..., y_K)$, the method considers the distributional uncertainty over permutations of candidate rankings within a TV distance $\rho$ of the observed ranking $\sigma^\star$. The robust objective is:

$$\ell_{rob}(\theta; x, Y, \sigma^\star) = (1-\rho)\ell_{PL}(\theta; x, Y, \sigma^\star) + \rho \max_{\sigma \in S_K} \ell_{PL}(\theta; x, Y, \sigma)$$

where $\ell_{PL}$ is the standard Plackett–Luce listwise loss.

Efficient Worst-case Adversary Construction

Naively, inner maximization over all $K!$ rankings entails combinatorial overhead. However, the structure of the PL loss enables an efficient solution: the adversarial worst-case ranking is achieved by sorting the implicit model scores in ascending order. With ties, any deterministic tie-breaking within tied groups is valid. Thus, the robust loss evaluation reduces to $O(K \log K)$ complexity per sample.

Theoretical Guarantees

Offline Fixed-list Setting

With a log-linear policy parameterization and a fixed candidate set, the robust PL loss is convex. The projected stochastic subgradient method reaches global $\epsilon$-suboptimality with $x$0 sample complexity—optimal for stochastic convex minimization. This is contrasted with prior robust DPO schemes (e.g., distributional robustness over prompts or pairs), where sample complexity can be significantly higher due to compounded uncertainty (Mandal et al., 1 Mar 2025, Xu et al., 4 Feb 2025).

Online Policy-induced Setting

When candidate lists are sampled on-policy from the current LLM, the objective is no longer convex but weakly convex. The paper provides a Clarke-subgradient oracle based on the ascending-sort solution and proves that projected stochastic subgradient updates enjoy $x$1 convergence to a stationary point of the Moreau envelope. All critical constants governing convergence are derived analytically in terms of explicit problem parameters.

Empirical Results

Offline Robustness Under Label Corruption

Experiments on the UltraFeedback dataset with $x$2 candidate lists focus on two classes of synthetic corruptions:

• Near-tie corruption: swaps between adjacent responses with similar scores, modeling annotation ambiguity.
• Top-rank corruption: promotes a lower-ranked candidate to the top, modeling more severe annotation errors.

The robust PL objective significantly improves Kendall's $x$3 rank correlation under top-rank corruption (e.g., for Qwen3-8B, $x$4 increases from 0.103 to 0.146 with full corruption), while incurring only modest degradation with clean data. This demonstrates that the method acts as a conservative regularizer, providing resilience against worst-case annotation noise without sacrificing performance in benign regimes.

Online Alignment With Reward Model Noise

In online alignment, LLMs generate response lists ranked by a fixed reward model, introducing realistic ranking-label uncertainty. Here, simply increasing candidate list size $x$5 without robustness yields inconsistent gains, as reward models are unreliable on finer-grained listwise comparisons. Applying the robust correction ($x$6) consistently stabilizes and improves reward model metrics, as well as external judge (GPT-4) evaluation scores. These gains hold for both small and intermediate LLMs.

Comparison to Baseline Robust Methods

Pairwise robust baselines (including TV-DR-DPO and KLDPO) do not match the robust PL's improvements under structured listwise corruption, and can overfit or degrade under excessive reweighting. This underscores the importance of genuinely listwise robustification.

Implications and Theoretical Contributions

This work advances the theoretical and practical understanding of robust preference optimization in LLM alignment:

• Conditional distributional robustness at the ranking-label level offers both mathematical tractability and direct mitigation of annotator/reward-model inconsistencies.
• Efficient and exact adversary solutions for worst-case listwise loss set a new standard for robust listwise learning-to-rank and alignment pipelines.
• Tight finite-sample complexity results in both convex (offline) and non-convex (online) regimes match or improve upon known rates for DPO-style robust optimization.
• Empirical evidence for practical robustness supports deployment in settings with unreliable annotations, especially as candidate sets and LLMs scale up.

Prospects for Future Work

Further research may focus on adaptive or data-driven selection of the robustness radius $x$7, leveraging more expressive or instance-dependent ambiguity sets, and stressing the theory and method in even larger-scale or more dynamic online deployments. Integration with advanced reward modeling and direct feedback aggregation from heterogeneous annotators are natural directions.

Conclusion

Distributionally robust listwise preference optimization, as developed in this work, provides a theoretically principled and computationally efficient framework for robust LLM alignment under realistic ranking-label uncertainty. It unifies listwise preference modeling and worst-case robustification with strong sample efficiency, and demonstrates substantial empirical resilience in scenarios with noisy feedback. The approach establishes a robust foundation for scalable, reliable LLM preference alignment and motivates further research at the intersection of learning-to-rank, distributionally robust optimization, and advanced human-AI interaction.