Structure from Strategic Interaction & Uncertainty: Risk Sensitive Games for Robust Preference Learning

Abstract: A growing line of work reframes preference-based fine-tuning of LLMs game-theoretically: Nash Learning from Human Feedback (NLHF) recasts the problem as a zero-sum game over policies. However, optimization is over expected pairwise payoffs, thereby conflating policies with similar win rates but different tail behavior. As such, these methods are agnostic to where in the data distribution they succeed or fail: strong average performance can mask systematic failure across prompts, annotators, or safety-critical strata. We introduce risk-sensitive preference games, in which players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. While risk-sensitivity generally breaks the zero-sum structure, we show that translation invariance of many risk metrics ensures that we retain monotonicity, yielding fast convergence of sample-efficient self-play methods. Furthermore, we establish algorithmic stability and offline sample complexity bounds that scale with risk, requiring simultaneous control of structural bias from nonlinear risk transformations, statistical bias in risk estimation, and concentration tailored to the risk-sensitive setting. To address statistical bias, we introduce a hierarchical game formulation and a two-timescale extragradient algorithm with bias correction that converges to the Stackelberg equilibrium and is especially effective in low-sample regimes. Empirically, risk-adjusted policies are robust across data strata, stable across risk choices, and match or exceed risk-neutral performance thereby achieving robustness without a performance tax.

• The paper introduces Risk Sensitive Preference Games that integrate convex risk measures to optimize beyond average performance, targeting tail risks.
• It proposes a novel two-timescale Stackelberg debiasing mechanism that reduces sample-induced bias, enhancing convergence and stability.
• Empirical results show risk-adjusted policies improve LLM robustness and safety across adversarial prompt strata without affecting average win rates.

Structure from Strategic Interaction and Uncertainty: Risk Sensitive Games for Robust Preference Learning

Overview and Motivation

This work addresses critical underexplored issues in the alignment of LLMs, specifically the failure of current RLHF and preference-based fine-tuning paradigms to provide robustness against systematic harms concentrated in the tails of the preference distribution. While recent advances such as Nash Learning from Human Feedback (NLHF) formulate alignment as a zero-sum preference game, these approaches optimize the expectation of the pairwise preference, causing policies with equivalent mean win-rates but highly divergent tail risks to be indistinguishable. This can result in policies that, while strong on average, systematically fail on certain user strata or adversarial prompt segments.

To overcome these limitations, the paper introduces Risk-Sensitive Preference Games (RSPG), embedding convex risk measures—derived from mathematical finance—within the preference-game framework. RSPGs enable optimization over features of the preference outcome distribution, such as variance or worst-case performance, rather than the mean alone, thereby allowing fine-grained control over robustness and safety-critical tails of the response distribution.

Technical Contributions

Risk-Sensitive Game Formulation

RSPG generalizes NLHF by replacing the expected payoff in the preference game with a convex risk measure $R$, operationalizing objectives such as entropic risk or CVaR. The resulting policies optimize

$$V_{\beta, R}(\pi_1, \pi_2) = \pi_i^\top P_R^i \pi_{-i} - \beta \mathrm{KL}(\pi_i\|\pi_\mathrm{ref}) + \beta \mathrm{KL}(\pi_{-i}\|\pi_\mathrm{ref})$$

where $P_R^i$ encapsulates the risk-adjusted preference operator for player $i$. This leads to the risk-adjusted quantal response equilibrium (RQRE), characterized by coupled fixed-point equations reflecting each player's best response under risk preference.

Preservation of Monotonicity and Computational Tractability

A principal technical challenge is that nonlinear risk transformations typically break the constant-sum structure that underlies the tractability of self-play algorithms for NLHF. The authors show that for translation-invariant convex risk measures (e.g., CVaR, entropic), the essential monotonicity and symmetry survive at the operator level, allowing the use of efficient single-policy self-play algorithms and retaining strong last-iterate convergence guarantees. The equilibrium of the risk-adjusted game remains unique and last-iterate convergence persists under standard variational inequality assumptions.

Statistical Bias and Two-Timescale Debiasing

Risk functionals, such as CVaR or entropic risk, lack unbiased sample gradients; plug-in estimation introduces a persistent bias that does not vanish with step size and is a fundamental bottleneck under limited sample budgets. The paper introduces a Stackelberg two-timescale extragradient mechanism in which a fast-moving follower tracks and subtracts the bias directly, allowing the leader to optimize a debiased objective. Theoretically, the residual error collapses from $O(1/m)$ (the irreducible bias) to $O(1/m^2)$ (the variance floor), and becomes exactly zero for certain risk measures (e.g., CVaR with the Rockafellar-Uryasev representation).

Generalization, Stability, and Offline Rates

The framework produces risk-adjusted equilibria with provable Lipschitz stability with respect to changes in the underlying risk-adjusted preference operator, enabling generalization bounds under both optimization and data-driven perturbations. The sample complexity for recovering the population RQRE matches the risk-neutral baseline up to explicit, quantified constants reflecting the degree of risk sensitivity.

Empirical Evaluation

Experiments on LLM safety alignment with PKU-SafeRLHF show that risk-adjusted policies dominate their risk-neutral counterparts not only in mean performance but also in the tails of the distribution, across diverse held-out prompt strata and safety-relevant categories. These gains are achieved without any tax to the average preference win-rate—risk-sensitive policies deliver uniformly improved robustness ratios and lower response variance. Notably, performance improvements in the deepest tails (e.g., worst 12.5% by CVaR) are significant and consistent across multiple data slices, confirming that robustness is not simply being reallocated from one region to another.

Cross-strata and cross-metric experiments demonstrate that risk-sensitive optimization fundamentally changes the robustness profile of the aligned model, maintaining stability on unseen conflict, severity, and ambiguous prompt segments. Alternative approaches, such as group DRO or re-weighting, fail to consistently realize the same robustness across these diverse axes.

Theoretical and Practical Implications

The integration of convex risk metrics into the strategic preference optimization setting advances the theoretical understanding of robustness in preference-based alignment:

• Unified Last-Iterate Convergence Theory: The translation-invariance result implies that convex risk transforms—although fundamentally nonlinear—preserve last-iterate self-play convergence under sufficient monotonicity, facilitating deployment at scale without loss of tractability.
• Quantified Robustness-Performance Tradeoffs: Degrees of risk sensitivity are tunable and entirely explicit in their impact on convergence and sample complexity, providing practitioners with direct control over the robustness/performance frontier.
• Remediation of Bias Barriers: The novel two-timescale Stackelberg correction addresses a critical statistical and optimization challenge in risk-sensitive learning, closing the previously irreducible gap in sample-limited regimes prevalent in LLM fine-tuning.
• Deployment-Ready Guarantees: Theoretically justified and empirically validated uniformity and stability of RQRE policies across heterogeneous and adversarial data strata support their use in high-risk real-world deployments, such as safety-critical LLMs.

Forward-Looking Discussion

The paper suggests multiple avenues for extensions:

• Beyond Monotone Regimes: While strong monotonicity ensures all desirable guarantees, analyzing algorithmic and statistical behavior in non-monotone (possibly cyclic) strategic risk games remains an open, high-impact direction.
• Population-Level and Performative Effects: Extending risk-sensitive learning to account for performative prediction—where the distribution of users, prompts, or annotators evolves with the deployed model—poses substantive modeling and algorithmic challenges.
• Stackelberg Debiasing for Broad Distributional Learning: The introduced two-timescale debiasing architecture generalizes to other distributionally robust equilibrium problems, including mean-field games and multi-agent reinforcement learning under risk preferences.

Conclusion

This paper reframes preference-based alignment as a problem of equilibrium over (possibly adversarial) preference outcome distributions, rather than merely optimizing the mean. Risk-sensitive preference games provide a principled, quantifiable, and computationally efficient mechanism for robust LLM alignment. Empirically, risk-sensitive policies consistently outperform or match risk-neutral baselines on both means and all relevant tails, establishing new state-of-the-art performance in alignment robustness without incurring a performance tax. This represents a substantive advance in the technical foundations and practical efficacy of robust learning from human feedback, with implications extending to any domain where robust population-level alignment is essential.

---

Reference:

"Structure from Strategic Interaction & Uncertainty: Risk Sensitive Games for Robust Preference Learning" (2605.09946)