---
title: Stackelberg Learning from Human Feedback
url: https://www.emergentmind.com/topics/stackelberg-learning-from-human-feedback-slhf
type: topic
---

# Stackelberg Learning from Human Feedback

Stackelberg Learning from Human Feedback (SLHF) is a sequential game-theoretic framework for preference optimization in machine learning, particularly for aligning large models with complex, often non-transitive and non-Markovian human preferences. SLHF decomposes the learning process into a leader–follower structure, where the Leader commits to an action or response and the Follower observes and conditionally refines or responds. This framework generalizes and improves upon classical Reinforcement Learning from Human Feedback (RLHF) and Nash Learning from Human Feedback (NLHF), offering significant benefits in alignment robustness, convergence properties, and inference-time adaptability [2512.16626].

## 1. Stackelberg Game-Theoretic Formulation

SLHF frames learning from preferences as a two-player, sequential-move (Stackelberg) game. Given a learned pairwise preference function
\[
p(y\succ y'\mid x) = \Pr(y \text{ preferred to } y' \mid x),
\]
where \(x\) is a context (e.g., prompt) and \(y, y'\) are candidate responses, two policies are defined:
- **Leader policy** \(\pi_L:\mathcal X \rightarrow \Delta_{\mathcal Y}\) samples an initial response \(y \sim \pi_L(\cdot\mid x)\).
- **Follower policy** \(\pi_F:\mathcal X\times\mathcal Y\rightarrow\Delta_{\mathcal Y}\) observes \(x, y\) and samples a refined response \(y' \sim \pi_F(\cdot\mid x, y)\).

The objective is
\[
J(\pi_L, \pi_F) = \mathbb{E}_{x\sim\rho}\left[
\mathbb{E}_{y\sim\pi_L, y'\sim\pi_F} [p(y\succ y'\mid x)]
+\tau^F \mathbb{E}_{y\sim\pi_L} [KL(\pi_F(\cdot|x, y) || \pi_F^{ref}(\cdot|x, y))]
-\tau^L KL(\pi_L(\cdot|x) || \pi_L^{ref}(\cdot|x))
\right]
\]
with \(\tau^F, \tau^L\) KL-regularization coefficients. The Stackelberg equilibrium requires solving
\[
\pi_L^* = \arg\max_{\pi_L} J(\pi_L, \pi_F^*(\cdot|\cdot, \cdot;\pi_L)),
\]
where \(\pi_F^*(\cdot|x, y;\pi_L)\) is the best response to the Leader, minimizing Follower loss conditional on the Leader's move [2512.16626].

## 2. SLHF Algorithms and Implementation

SLHF equilibrium is typically approximated via two-timescale gradient descent–ascent (2×GDA):
1. Sample \(\{x_i\}\) from the context distribution.
2. For each \(x_i\), generate Leader sample \(y_i \sim \pi_L(\cdot|x_i)\) and Follower sample \(y'_i \sim \pi_F(\cdot|x_i, y_i)\).
3. Compute preference scores \(p_i = p(y_i \succ y'_i | x_i)\).
4. Update policy parameters with gradients:
   \[
   \widehat\nabla_\theta J = \frac{1}{B}\sum_i (p_i - \tau^L k_i^L)\nabla_\theta \log\pi_\theta(y_i|x_i)
   \]
   \[
   \widehat\nabla_\phi J = \frac{1}{B}\sum_i (p_i - \tau^F k_i^F)\nabla_\phi \log\omega_\phi(y'_i | x_i, y_i)
   \]
   where \(k_i^L, k_i^F\) are ratios to the reference policies.
5. Parameters are stepped with learning rates \(\alpha^L \ll \alpha^F\).

When implementing on transformer architectures, the Leader and Follower can be collapsed into a single network with separate prompt templates. The two-timescale loss is composed of respective log-probability terms for Leader and Follower outputs [2512.16626].

In the RL setting with trajectory preference oracles, SLHF may be realized as a self-play algorithm where empirical win-rates over pairs of policy samples serve as non-scalar, trajectory-dependent rewards. Policy update proceeds via off-the-shelf RL optimizers (e.g., PPO, SAC) using batchwise self-comparisons, admitting robust credit assignment even for non-Markovian, non-transitive, or stochastic preference feedback [2401.04056].

## 3. Comparison with RLHF and Nash-Equilibrium Approaches

The relationship between SLHF, RLHF, and NLHF can be summarized as follows:

| Method | Solution Concept | Preference Assumptions | Typical Equilibrium |
|--------|-----------------|-----------------------|--------------------|
| RLHF   | Reward maximization via Bradley–Terry or similar reward modeling | Requires scalarizable, transitive preferences | Deterministic if consistent, but fails for cycles/intransitivity |
| NLHF   | Simultaneous-move zero-sum (Nash) game | Arbitrary pairwise preferences (concave–convex) | Mixed, high entropy; symmetric strategies |
| SLHF   | Sequential-move Stackelberg game | Arbitrary, possibly intransitive/non-Markovian | Often deterministic; robust under cycles/intransitivity [2512.16626, 2401.04056] |

A key advantage of SLHF is its handling of intransitive preference structures. In Condorcet cycle scenarios, RLHF can become unstable or arbitrarily select among cycle participants, and NLHF equilibria degenerate to uniform mixtures. SLHF recovers robust (Condorcet-winner-like) policies by leveraging the Follower’s sequential conditionality and the Leader’s anticipation of best-response refinement [2512.16626].

Additionally, RLHF is sensitive to missing pairwise comparison data; omitted pairs can cause the learned argmax to shift unpredictably. SLHF's solution is dataset-independent in that the Leader's optimal policy is not acutely dependent on the annotated pairs [2512.16626].

## 4. Robustness, Preference Structure, and Theoretical Guarantees

SLHF is specifically robust to the following phenomena:
- **Non-Markovian preferences**: The preference function may depend on entire trajectories or sequences, not just single-step states or actions. SLHF operates without fitting Markovian reward models, so path-dependent or history-sensitive feedback does not degrade policy learning [2401.04056].
- **Intransitive preferences or cycles**: MW (minimax-winner) aggregation and Stackelberg game-theoretic structure absorb cycles (e.g., \(A \succ B \succ C \succ A\)) via mixed or iterative policies [2512.16626].
- **Stochastic/Noisy feedback**: Because rewards are empirical win-rates derived from preference queries among batches, stochastic fluctuations are averaged out, and no adversarial reward fitting can amplify label noise [2401.04056].

Theoretical guarantees follow from online learning and no-regret analysis. In single-agent self-play, an average policy \(\bar\pi\) after \(T\) iterations with regret \(\mathrm{Reg}(T)\) satisfies the minimax-winner property:
\[
\max_{\pi}\min_{\pi'} A(\pi, \pi') - \min_{\pi'}A(\bar\pi, \pi') \leq \tfrac{2\mathrm{Reg}(T)}{T}
\]
For no-regret oracles with \(\mathrm{Reg}(T)=O(\sqrt{T})\), this yields convergence at \(O(1/\sqrt{T})\); stronger guarantees (\(O(1/T)\)) are available under unique gap assumptions [2401.04056].

In Markov games with quantal-response models, regret bounds inherit the dependence on model class dimension and interaction horizon \(H\). Linear/myopic approximations enable \(\tilde{O}(H^2 d\sqrt{T})\) regret with efficient updates [2307.14085].

## 5. Inference-Time Refinement and Deployment Advantages

SLHF uniquely enables inference-time refinement via iterative application of the Follower policy:
1. Sample initial response \(y_1 \sim \pi_L^*(\cdot|x)\).
2. For \(i>1\), sample iteratively \(y_i \sim \pi_F^*(\cdot|x, y_{i-1})\).
This produces a chain of refined responses that efficiently traverses high-preference regions of response space.

In environments with cycles or substantial intransitivities, this method guarantees coverage—e.g., for a Condorcet cycle with options \(\{A, B, C\}\), a sufficiently long chain will expose all underlying user preferences within a bounded number of steps. Unlike RLHF or NLHF, which rely on i.i.d. sampling, SLHF's sequential sampling structure ensures both convergence and adaptability to individual user feedback [2512.16626].

Moreover, the Follower policy is transferable: it can refine outputs of models other than those seen during its own training, without fine-tuning. This property has been empirically validated on diverse generative models including Qwen2.5 and Llama-3.1 [2512.16626].

## 6. Empirical Results and Performance Scaling

Empirical studies across models from 0.5B to 8B parameters and datasets like HelpSteer2 (11.8K prompts, five annotated attributes) establish that SLHF consistently outperforms both RLHF variants (RLOO) and Nash-equilibrium NLHF methods (Nash-MD):
- **SLHF Leader**: Achieves ≈73% win-rate vs. baseline; Follower refinements further boost performance to ≈80% [2512.16626].
- **SLHF Follower used on other models**: Provides >60% improvements in win-rate relative to those models’ baselines.
- **Scaling**: SLHF retains improvements (~80% win vs. baseline) at 1.5B and 3B model sizes; Leader performance lags at the largest model sizes but improves with longer training.
- **Robustness**: In HelpSteer2, ~57% of prompts induce intransitive cycles among outputs; SLHF handles these naturally, while RLHF and NLHF show degraded or unstable performance.
- **Inference-time N-best scaling**: SLHF's preference scores increase with \(N\leq 5\), in contrast to RLHF/NLHF collapse or early saturation.

On open-domain chat (Llama-3.1-8B), SLHF yields AlpacaEval 2.0 win-rates of ≈48% (vs. 44% DPO, 35% RLOO) and IFEval win-rates of ≈71% (compared to 62% DPO and 72% RLOO), indicating a preference–QA tradeoff [2512.16626]. 

## 7. Extensions, Model Structure, and Future Directions

SLHF generalizes to Markov games and settings involving strategic human feedback via quantal Stackelberg equilibria. The human is modeled as a quantally rational follower, with a temperature parameter estimated via maximum-likelihood from observed actions [2307.14085]. Model-uncertainty can be managed through confidence sets and pessimistic/optimistic planning (offline/online), with regret bounds scaling in dimension and data as dictated by the underlying Bellman and quantal-response error classes.

For richer feedback—e.g., multi-step value disclosures or pairwise comparison of complete trajectories—the core MLE fitting algorithms extend straightforwardly by adding likelihood terms.

A plausible implication is that SLHF's sequential, anticipative structure provides a general foundation for constructing robust, user-adaptive, high-capacity models in domains where preferences are complex, intransitive, or contextually entangled, and where inference-time refinements are essential for personalized alignment.

---

**Principal References**: [2512.16626], [2401.04056], [2307.14085]

Source: https://www.emergentmind.com/topics/stackelberg-learning-from-human-feedback-slhf