Multiplayer Nash Preference Optimization (2509.23102v1)

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning LLMs with human preferences. However, reward-based methods built on the Bradley-Terry assumption struggle to capture the non-transitive and heterogeneous nature of real-world preferences. To address this, recent studies have reframed alignment as a two-player Nash game, giving rise to Nash learning from human feedback (NLHF). While this perspective has inspired algorithms such as INPO, ONPO, and EGPO with strong theoretical and empirical guarantees, they remain fundamentally restricted to two-player interactions, creating a single-opponent bias that fails to capture the full complexity of realistic preference structures. In this work, we introduce Multiplayer Nash Preference Optimization (MNPO), a novel framework that generalizes NLHF to the multiplayer regime. It formulates alignment as an $n$-player game, where each policy competes against a population of opponents while being regularized toward a reference model. Our framework establishes well-defined Nash equilibria in multiplayer settings and extends the concept of duality gap to quantify approximation quality. We demonstrate that MNPO inherits the equilibrium guarantees of two-player methods while enabling richer competitive dynamics and improved coverage of diverse preference structures. Through comprehensive empirical evaluation, we show that MNPO consistently outperforms existing NLHF baselines on instruction-following benchmarks, achieving superior alignment quality under heterogeneous annotator conditions and mixed-policy evaluation scenarios. Together, these results establish MNPO as a principled and scalable framework for aligning LLMs with complex, non-transitive human preferences. Code is available at https://github.com/smiles724/MNPO.

• The paper introduces MNPO, a multiplayer extension of Nash learning that enables simultaneous policy competition for capturing heterogeneous, non-transitive human preferences.
• It employs an iterative, multiplicative-weights update inspired by online mirror descent, integrating historical and external policy information to stabilize training.
• Empirical results demonstrate MNPO outperforms existing methods in instruction-following, reasoning, and coding tasks, reinforcing its robust multi-agent alignment strategy.

Multiplayer Nash Preference Optimization: A Game-Theoretic Framework for LLM Alignment

Motivation and Background

Reinforcement learning from human feedback (RLHF) has become the standard paradigm for aligning LLMs with human preferences. Traditional RLHF methods, such as those based on the Bradley-Terry model, assume transitive and homogeneous preferences, which are often violated in real-world settings where annotator judgments are heterogeneous and non-transitive. Recent advances have reframed preference optimization as a two-player Nash game, leading to Nash learning from human feedback (NLHF) and algorithms such as INPO, ONPO, and EGPO. However, these approaches are fundamentally limited to two-player interactions, introducing a single-opponent bias and failing to capture the complexity of realistic, multi-annotator or multi-policy preference structures.

Theoretical Framework

The paper introduces Multiplayer Nash Preference Optimization (MNPO), which generalizes NLHF to the $n$-player regime. In MNPO, each policy in a population simultaneously competes against all other policies while being regularized toward a reference model. The objective for each policy $\pi_i$ is to maximize its expected preference probability over the responses of all other policies $\{\pi_j\}_{j \neq i}$, subject to a KL regularization term that penalizes deviation from a reference policy $\pi_{\text{ref}}$:

$$J\left(\pi_i,\left\{\pi_j\right\}_{j \neq i}\right) = \mathbb{E}_{x \sim D}\left[\mathbb{E}_{y^i \sim \pi_i, \{y^j \sim \pi_j\}_{j \neq i}}\left[\mathbb{P}\left(y^i \succ \{y^j\}_{j \neq i} \mid x\right)\right] - \tau \mathrm{KL}\left(\pi_i(\cdot \mid x) \| \pi_{\mathrm{ref}}(\cdot \mid x)\right)\right]$$

This formulation admits well-defined Nash equilibria in the multiplayer setting, with the equilibrium policy $\pi^*$ satisfying that no player can improve their objective by unilaterally deviating. The duality gap is extended to quantify the approximation quality of a policy relative to the Nash equilibrium, and the equilibrium win rate generalizes to $1/n$ for $n$ players.

The framework supports both the Plackett-Luce reward learning assumption (for listwise comparisons) and a general preference oracle, allowing for flexible modeling of complex, non-transitive, and heterogeneous preference structures.

Algorithmic Contributions

MNPO is instantiated via an iterative, multiplicative-weights update inspired by online mirror descent. At each iteration, each policy $\pi_i$ is updated according to the average advantage over all other policies, with the update rule:

$$\pi_i^{(t+1)}(y \mid x) \propto \left(\prod_{j \neq i} \pi_j^{(t)}(y \mid x)\right)^{\frac{1}{n-1}} \exp\left(\frac{\eta}{n-1} \sum_{j \neq i} \mathbb{P}\left(y \succ \pi_j^{(t)} \mid x\right)\right)$$

This update ensures that responses with higher average advantage over the population are upweighted, while the geometric mean over opponents stabilizes the update. The loss function for policy optimization is constructed to avoid intractable normalization over the response space, relying instead on pairwise log-ratio dynamics.

A key innovation is the introduction of time-dependent MNPO (TD-MNPO), where the set of opponents is constructed as a weighted mixture of historical policies, enabling the model to incorporate past knowledge and stabilize training. The framework also supports the use of external LLMs as opponents (EO-MNPO), generalizing knowledge distillation and enabling robust optimization against diverse policy populations.

MNPO unifies a broad family of preference optimization algorithms (e.g., DPO, SimPO, INPO, SPPO) as special cases, depending on the number of players, opponent selection, and loss formulation.

Empirical Evaluation

MNPO is evaluated on a suite of instruction-following and reasoning benchmarks, including AlpacaEval 2.0, Arena-Hard, and MT-Bench, as well as academic benchmarks covering instruction following, knowledge, commonsense reasoning, mathematics, and coding. The experiments use Gemma-2-9B-it as the base model, with preference signals provided by a strong reward model (ArmoRM-Llama3-8B-v0.1).

Key empirical findings:

• Instruction-following: MNPO achieves a score of 57.27 on AlpacaEval 2.0, outperforming DPO (54.35), SimPO (55.16), SPPO (55.97), and INPO (56.09). On Arena-Hard, MNPO achieves 52.26, a 4.23-point improvement over INPO (48.03), and surpasses much larger open-source and closed-source models, including GPT-5.
• Academic benchmarks: MNPO attains the highest average score (71.08) across instruction, knowledge, and commonsense tasks, and demonstrates strong performance on graduate-level reasoning (GPQA: 33.33).
• Mathematics and coding: MNPO is the only method to achieve non-zero performance on AIME-24 (3.33), and achieves the best coding performance on HumanEval (61.59).

These results demonstrate that the multiplayer formulation of MNPO provides significant advantages in aligning LLMs with heterogeneous and non-transitive human preferences, while preserving or improving performance on reasoning and factual tasks.

Practical and Theoretical Implications

MNPO addresses the limitations of two-player preference optimization by enabling richer competitive dynamics and improved coverage of diverse preference structures. The multiplayer game-theoretic formulation is particularly well-suited for scenarios involving multiple annotators, heterogeneous evaluation criteria, or mixtures of historical and external policies. The iterative, population-based update mechanism provides stable convergence guarantees and mitigates overfitting to transient fluctuations.

The framework's unification of existing preference optimization methods under a single, principled objective facilitates systematic comparison and adaptation to different alignment scenarios. The ability to incorporate external LLMs as opponents opens new avenues for robust knowledge integration and domain adaptation.

From a theoretical perspective, MNPO extends the equilibrium and duality gap concepts to the multiplayer regime, providing a foundation for future work on scalable, robust, and generalizable alignment algorithms.

Limitations and Future Directions

The performance of MNPO is fundamentally constrained by the quality of the preference oracle. As models improve and the preference gap narrows, binary preference signals become less informative, potentially slowing convergence. Future research should explore more nuanced feedback mechanisms, such as graded or multi-dimensional preferences, and investigate the integration of richer reward signals.

The extension to external opponent policies suggests promising directions for multi-agent RL and knowledge distillation, enabling alignment across diverse model families and domains. Further work is needed to characterize the theoretical properties of MNPO in non-stationary or adversarial settings, and to scale the approach to larger policy populations and more complex preference structures.

Conclusion

Multiplayer Nash Preference Optimization generalizes Nash learning from human feedback to the multiplayer setting, providing a principled, scalable, and robust framework for aligning LLMs with complex, heterogeneous, and non-transitive human preferences. The framework unifies and extends existing preference optimization methods, achieves strong empirical results across a range of benchmarks, and opens new directions for research in game-theoretic alignment and multi-agent learning.