Nash Learning from Human Feedback (NLHF)

• NLHF is a game-theoretic paradigm that reframes LLM alignment as a search for Nash equilibrium through pairwise human preference data.
• It employs maximal lotteries and mixed strategies to naturally handle cyclic and non-transitive preferences, ensuring fair representation of minority views.
• Various algorithms, such as mirror descent and extragradient methods, demonstrate robust convergence and bias-free performance under complex preference scenarios.

Nash Learning from Human Feedback (NLHF) is a game-theoretic paradigm for aligning LLMs with human preferences. It reframes policy optimization as the search for a Nash equilibrium in a two-player zero-sum game defined by pairwise preference data, rather than relying on reward models. NLHF addresses key statistical and structural limitations of standard reinforcement learning from human feedback (RLHF), notably robustness to non-transitive and heterogeneous preferences, and enables provable preservation of minority and cyclic preferences.

1. Foundational Formulation and Statistical Motivation

NLHF formalizes the model alignment problem as follows. Let $\mathcal{Y} = \{y_1, \ldots, y_n\}$ denote the set of candidate responses for a prompt $x$. Human feedback is collected as pairwise preference probabilities $\mathcal{P}(y \succ y'\mid x)$, specifying the probability that $y$ is preferred over $y'$. Two policies, $\pi$ and $\pi'$, each specifying a distribution over $\mathcal{Y}$, define the moves of Player 1 and Player 2 in a zero-sum game, with the payoff function: $$u(\pi, \pi') = \mathbb{E}_{y \sim \pi, y' \sim \pi'}[\mathcal{P}(y \succ y')]$$ The NLHF objective seeks the Nash equilibrium: $\max_\pi \min_{\pi'} \ u(\pi, \pi').$

With high probability, under the impartial culture model (random labeler preferences), preference graphs exhibit Condorcet cycles as the number of responses $n$ increases, and can therefore not be faithfully modeled by any scalar reward function (Liu et al., 14 Mar 2025). This renders reward-based RLHF approaches statistically insufficient for general human alignment, as they cannot capture the prevalence of cyclic or intransitive preferences and will fail to represent minority groups fairly.

Crucially, in the absence of a Condorcet winner, all Nash equilibria in NLHF must be mixed (having support on multiple responses), ensuring stable preservation of diverse, non-majoritarian preferences (Liu et al., 14 Mar 2025). The Nash framework thereby naturally encodes the probabilistic aggregation of conflicting preferences in the output policy.

2. Game-Theoretic Structure: Maximal Lotteries and Social Choice

NLHF instantiates the concept of maximal lottery from probabilistic social choice (Maura-Rivero et al., 31 Jan 2025). The mixed-strategy Nash equilibrium corresponds precisely to the maximal lottery rule, which offers key axiomatic properties:

• Majority consistency: If a Condorcet-winning response exists, it is deterministically selected.
• Condorcet and cycle handling: In the case of cycles, mass is distributed over the entire Smith set in a way determined by the population preferences.
• Independence of Irrelevant Alternatives (IIA): Marginals remain stable under the addition/removal of irrelevant alternatives.

These properties are rigorously proven: the Nash (max–min) solution coincides with the maximal-lottery policy, and thus inherits majority, Condorcet, and IIA properties (Maura-Rivero et al., 31 Jan 2025). This addresses documented failures of RLHF, such as Borda count bias and mode collapse under cyclic feedback.

3. Core Algorithms for Nash Equilibrium Computation

Multiple algorithmic frameworks instantiate NLHF in practice, adapted to both tabular and parametric (deep neural) policy classes:

Algorithm	Optimization Principle	Theoretical Guarantee (Regularized/Unregularized)
Nash-MD	Mirror Descent	$O(1/T)$ KL last-iterate in tabular setting (Munos et al., 2023)
Nash-MP	Mirror Prox	Linear ($O(\rho^T)$) last-iterate in KL (Tiapkin et al., 26 May 2025)
EGPO	Extragradient	Linear last-iterate (KL-reg.), polynomial for unreg. (Zhou et al., 11 Mar 2025)
OMWU	Optimistic Mult. Weights	Linear last-iterate, unregularized NE (Chen et al., 31 Dec 2025)
INPO/ONPO	Online/Optimistic Mirror Descent	Convergence of average to NE at $O(1/\sqrt{T})$/$O(1/T)$
SPO	Self-Play Preference Optimization	No-regret to Minimax Winner; $O(1/T)$ rate with gap (Swamy et al., 2024)

Several approaches, such as Nash-RS, employ efficient single-loop approximate maximization via stochastic policy gradients and rejection sampling (Liu et al., 14 Mar 2025). All guarantee that, under mild regularity, the sequence (or average) of policies converges to a Nash equilibrium under policy-level preference maximization, even when preferences are non-transitive or cyclic.

4. Extensions: Multiplayer, Multi-turn, and Multi-agent NLHF

Recent work generalizes NLHF beyond the two-player, single-turn case:

Multiplayer Nash Preference Optimization (MNPO): Extends to an $n$-player game where each policy plays against a population of opponents, rather than a single adversary (Wu et al., 27 Sep 2025). The objective is generalized to

$$J_i(\pi_i, \pi_{-i}) = \mathbb{E}_{x}\left[ \mathbb{E}_{y^i, \{y^j\}_{j \ne i}} [\mathcal{P}(y^i \succ \{y^j\}_{j \ne i} | x)] \right] - \tau \mathrm{KL}(\pi_i || \pi_{\rm ref}),$$

establishing existence and convergence of Nash equilibria and improved robustness to heterogeneous annotator populations.

Multi-turn NLHF: Extends the Nash equilibrium concept to trajectory-level preferences in finite-horizon contextual MDPs, with policy optimization via regularized mirror descent on preference-based Q-functions (Shani et al., 2024). Empirical results show that trajectory-level NLHF outperforms both single-turn and standard multi-turn RLHF baselines in environments where reward-based modeling is insufficient.

Multi-agent Preference-Based MARL: Identifies that single-policy coverage is insufficient for Nash recovery in offline multi-agent RL; efficient recovery requires unilateral coverage. Regularization and pessimism techniques are introduced for learning from sparse offline preference data (Zhang et al., 2024).

5. Practical Implementation and Empirical Performance

Empirical validations demonstrate that NLHF-based fine-tuning improves win rates and mean preference scores over conventional RLHF and DPO baselines across a range of tasks and evaluation suites, including Llama-3 and Gemma models, AlpacaEval, Arena-Hard, MT-Bench, and complex academic benchmarks (Tiapkin et al., 26 May 2025, Liu et al., 14 Mar 2025, Wu et al., 27 Sep 2025). NLHF methods, especially those employing Mirror Prox, Extragradient, or OMWU, realize last-iterate, bias-free convergence to Nash equilibrium policies and outperform both regularized and unregularized variants of mirror descent and online policy optimization.

Best practices include single-loop algorithms (avoiding nested optimization), use of entropy or explicit KL regularization to stabilize and control proximity to pre-trained policies, rejection sampling or preference regression losses, and maintaining exploration via entropy bonuses or replay (Liu et al., 14 Mar 2025, Swamy et al., 2024). Robustness to noise, preference misspecification, and sample complexity scaling are active areas of research.

6. Limitations, Open Challenges, and Future Directions

Identified limitations and open problems include:

• Non-uniqueness and convergence: For non-full-support or degenerate preference matrices, more precise last-iterate guarantees and optimal sample complexity remain open.
• Pairwise preference data: Annotation cost, coverage, and active data selection strategies are open for large-scale deployments.
• Extension to full dialogue or compositional tasks: While multi-turn NLHF is theoretically grounded (Shani et al., 2024), practical LLM alignment across extended interactions is ongoing.
• Diversity and fairness: Explicit regularizers for diversity or mechanisms for weighting minority preferences can be combined with NLHF for refined control (Liu et al., 14 Mar 2025).
• Scale-sensitive self-alignment: Algorithms such as LANA demonstrate self-alignment, but rely on initial model quality and require further scalability studies (Azarafrooz et al., 2024).

7. Summary of Significance

NLHF constitutes a robust, theoretically principled, and empirically validated alternative to reward-based RLHF for aligning LLMs with diverse, non-transitive, and population-scale human preferences. By solving for the Nash equilibrium of the induced preference game, NLHF guarantees probabilistic fairness, resilience to preference cycles, and protection against exploitation or mode collapse, all with strong convergence properties under algorithmic variants such as mirror descent, mirror prox, extragradient, and optimistic multiplicative weights. These advancements position NLHF as a foundational framework for next-generation fair and robust LLM alignment (Liu et al., 14 Mar 2025, Munos et al., 2023, Tiapkin et al., 26 May 2025, Wu et al., 27 Sep 2025, Swamy et al., 2024, Chen et al., 31 Dec 2025).