MaxEntBW: Maximum Entropy Blackwell Winner

• The paper demonstrates how MaxEntBW introduces entropy regularization to yield uniquely defined policies under intransitive multi-objective preferences.
• It employs a three-player zero-sum game formulation that reduces to a tractable concave maximization, enhancing both optimization efficiency and theoretical soundness.
• Empirical results using the PROSPER algorithm show improved win-rates and robust fine-tuning performance for large language models in varied settings.

The Maximum Entropy Blackwell Winner (MaxEntBW) is a game-theoretic solution concept for preference fine-tuning (PFT) problems characterized by multi-objective and intransitive (cyclic) preferences. MaxEntBW arises in settings where rankings on model outputs are inconsistent—stemming either from conflicting objectives or from attempts to scalarize multiple objectives—which preclude the existence of a well-defined optimal policy under classical minimax or Condorcet approaches. MaxEntBW addresses these deficits by formulating the PFT problem as a three-player zero-sum game and introducing entropy regularization (via Kullback–Leibler divergence), resulting in a robust, uniquely defined policy for complex preference structures (Zhang et al., 22 Feb 2026).

1. Formal Definition and Mathematical Structure

Let $X$ denote the prompt space, $Y$ the response space, and $\Pi \subset \{X \rightarrow \Delta(Y)\}$ a convex, compact set of randomized policies. For each prompt $x$, a multi-objective judge produces a vector $P(y \succ y'|x) \in [0,1]^{m(x)}$, evaluating $m(x)$ criteria. For policies $\pi, \pi'$, expected pairwise preference is

$$P(\pi \succ \pi' | x) = \mathbb{E}_{y\sim\pi(x), y'\sim\pi'(x)} \left[ P(y \succ y'|x) \right] \in [0,1]^{m(x)}.$$

The MaxEntBW framework sets up a three-player zero-sum game at each $x$ involving: - The Learner ($\pi \in \Pi$) - The Objective-chooser ($w(x) \in \Delta_{m(x)}$) - The Adversary ($\pi' \in \Pi$)

The instantaneous payoff is:

$$\langle w(x), P(\pi \succ \pi'|x) \rangle + \beta\, D_{\mathrm{KL}}(\pi'(x)\|\pi_{\text{ref}}(x)),$$

where $\pi_{\text{ref}}$ is a fixed reference policy and $\beta > 0$ controls adversarial regularization. The learner’s value is:

$$V(\pi) = \min_{w: X \rightarrow \Delta_{m(x)}} \min_{\pi' \in \Pi} \mathbb{E}_{x \sim \mu}\left[ \langle w(x), P(\pi \succ \pi'|x) \rangle + \beta\, D_{\mathrm{KL}}(\pi'(x)\| \pi_{\text{ref}}(x)) \right].$$

A Maximum Entropy Blackwell Winner is any $\pi^* \in \arg\max_{\pi \in \Pi} V(\pi)$.

2. Game-Theoretic and Operational Intuition

Standard preference tuning familiar from single-objective settings (Condorcet winners, von Neumann minimax) assumes transitive (acyclic) preferences. When preferences are intransitive—typical in real-world, multi-criteria evaluation—no global ranking or optimal policy exists. The multi-objective extension, inspired by Blackwell’s approachability, seeks policies whose win-rate vectors approach a desirable set (e.g., all coordinates $\geq 1/2$). The classical Blackwell Winner solves

$$\operatorname{BW} = \arg\min_{\pi} \max_{\pi'} \mathrm{dist}_\infty(P(\pi \succ \pi'),\,C),$$

where $C$ is the desirable set. However, this solution, without regularization, requires solving a challenging adversarial self-play game and is susceptible to nonconvexities.

MaxEntBW modifies Blackwell’s formulation through three core steps: - KL-regularization anchors the adversary to $\pi_{\text{ref}}$, ensuring unique solutions and improved optimization properties. - Order-of-play swap preserves game well-posedness under conflicting objectives. - Reduction to a concave maximization uses the explicit solution of the inner minimax layer, collapsing the three-player game to a tractable single-player problem.

3. Optimization Framework and Theoretical Properties

The computation of MaxEntBW policies exploits the convex-analytic structure introduced by regularization. For fixed $\pi$ and $w$, the adversarial minimization

$$\min_{\pi' \in \Pi} \mathbb{E}_x\left[ \langle w(x), P(\pi \succ \pi'|x) \rangle + \beta\,D_{\mathrm{KL}}(\pi'(x)\| \pi_{\text{ref}}(x)) \right]$$

admits a unique solution,

$$\pi'_\star(y'|x) = \frac{\pi_{\text{ref}}(y'|x) \exp \left( - \langle w(x), P_\pi(y'|x) \rangle / \beta \right)} {Z(w, \pi|x)},$$

with $$P_\pi(y'|x) = \mathbb{E}_{y \sim \pi(x)} [P(y \succ y'|x)]$$ and normalization $Z(w, \pi|x)$. Substituting this yields the two-player problem:

$$\max_\pi \min_{w(\cdot)} \mathbb{E}_x\left[ -\beta \log Z(w,\pi|x) \right].$$

Crucially, the minimization over $w(x) \in \Delta_{m(x)}$—a simplex—localizes to a vertex, so the complexity reduces to coordinate-wise evaluation:

$$\min_{w(x)} \,\cdots = \min_{k \in [m(x)]}\left[ -\beta \log Z^k(\pi|x) \right],$$

where $$Z^k(\pi|x) = \mathbb{E}_{y' \sim \pi_{\text{ref}}(x)} \left[ \exp \left( -P_\pi^k(y'|x)/\beta \right) \right]$$. The final learning objective is a single-player, concave maximization:

$$\max_{\pi \in \Pi} \mathbb{E}_x \left[ -\beta \log Z^{k^*(x)}(\pi|x) \right],$$

where $k^*(x) = \arg\min_k[-\beta \log Z^k(\pi|x)]$.

This objective is concave in $\pi$ due to the pointwise minimization over concave functions and the concavity of each $-\beta \log Z^k$.

4. The PROSPER Algorithm

PROSPER (Policy Regularized Optimization for Scalarized, Preference-Enabled Reward) is a scalable implementation for computing approximate MaxEntBW policies in large model spaces. PROSPER leverages stochastic mirror-descent with a KL-divergence Bregman potential, implemented using regression-based gradient estimation (REBEL/ReF-MART methodology).

PROSPER algorithm workflow:

• Initialize $\theta_0$ to parameterize $\pi_{\theta_0} = \pi_{\text{ref}}$.
• At each timestep $t$:
  • Sample prompts $x$ from dataset $D$.
  • Generate $M$ responses each from $\pi_{\theta_t}(x)$ and $\pi_{\text{ref}}(x)$.
  • For each $x$ and objective index $k$:
  • Estimate $Z^k(x)$ via mini-batch MC sampling.
  • Identify $k^*(x)$ as minimizing $-\beta \log Z^k(x)$.
  • For sampled responses $z$ (on-policy) and $z'$ (off-policy), compute gradient weights $\hat{g}_t(x, z)$ with targeted importance weighting over the preference scores.
  • Update parameters via squared-loss regression to match the policy gradient implied by the estimated weights.

Each iteration requires $O(|D|M^2)$ judge calls and a single regression epoch. There is no adversarial inner loop at runtime.

Theoretical guarantee: Under mild concentrability and regression-error ($\varepsilon$) assumptions, after $T$ steps, PROSPER achieves $$V(\pi^*) - V(\hat{\pi}) \leq O(1/\sqrt{T} + \sqrt{C_{\pi_{\text{ref}} \to \pi^*} \varepsilon})$$, where $C$ encodes the concentrability between policies (Zhang et al., 22 Feb 2026).

5. Empirical Performance in LLM Fine-Tuning

PROSPER has been applied to instruction-tuned LLMs (Qwen2.5-Instruct at 3B and 7B scale), evaluated with prompt-specific checklists ("WildChecklists") for multi-objective feedback. The judge is a Qwen3-14B LLM, returning per-objective preference scores.

Comparison baselines include:

• RLCF: Scalarizes rubric scores via LLM-generated weights plus RL.
• PROSPER-JC: Collapses all objectives to a joint scalar score ($m=1$).
• PROSPER-VB: Omits adversarial $\pi'$, effectively $\beta \to \infty$.

Empirical results at 7B scale show:

• PROSPER attains the highest win-rates on in-domain alignment benchmarks (AlpacaEval 2.0, Arena-Hard; e.g., 49.2% win vs. 42.4% baseline on Arena-Hard).
• In pairwise win-rate matrices, PROSPER beats the base model approximately three-quarters of the time, and RLCF about two-thirds.
• Performance on out-of-domain QA and reasoning (MMLU, ARC, HellaSwag, TruthfulQA) is maintained or slightly improved.
• Ablations underperform, underscoring the importance of both adversarial modeling and explicit multi-objective handling.

6. Theoretical Insights and Structural Properties

Several key theoretical properties underlie MaxEntBW and PROSPER:

• Each mapping $\pi \mapsto -\beta \log Z^k(\pi|x)$ is concave, and pointwise minima over $k$ preserve concavity (Lemma 3.2).
• The minimization over $w(x) \in \Delta_{m(x)}$ reduces to a vertex of the simplex by Bauer's Maximum Principle.
• KL-regularization yields a closed-form Gibbs minimizer for the adversary, eliminating the need for explicit adversarial search in optimization.
• The mirror-descent update with regression approximates the optimal policy, converging at rate $O(1/\sqrt{T})$ (Theorem 4.1).

A plausible implication is that these properties substantially improve the tractability and robustness of multi-objective PFT, especially in the presence of intransitive or cyclic preference feedback.

7. Significance and Implications

MaxEntBW constitutes a robust extension of Blackwell’s approachability to high-dimensional, intransitive preference optimization, overcoming limitations of classical minimax self-play and scalarization-based RL techniques. By providing a well-posed, single-policy objective even under conflicting multi-objective signals, MaxEntBW—exemplified via the scalable PROSPER algorithm—enables efficient and theoretically grounded fine-tuning of large models using rubric-based, multi-objective evaluators. Empirical evidence suggests this approach yields improved alignment and generalization, especially in settings where scalarization and standard PFT pipelines fail due to cyclic preferences (Zhang et al., 22 Feb 2026).