Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning

Abstract: A recurring challenge in preference fine-tuning (PFT) is handling $\textit{intransitive}$ (i.e., cyclic) preferences. Intransitive preferences often stem from either $\textit{(i)}$ inconsistent rankings along a single objective or $\textit{(ii)}$ scalarizing multiple objectives into a single metric. Regardless of their source, the downstream implication of intransitive preferences is the same: there is no well-defined optimal policy, breaking a core assumption of the standard PFT pipeline. In response, we propose a novel, game-theoretic solution concept -- the $\textit{Maximum Entropy Blackwell Winner}$ ($\textit{MaxEntBW}$) -- that is well-defined under multi-objective intransitive preferences. To enable computing MaxEntBWs at scale, we derive $\texttt{PROSPER}$: a provably efficient PFT algorithm. Unlike prior self-play techniques, $\texttt{PROSPER}$ directly handles multiple objectives without requiring scalarization. We then apply $\texttt{PROSPER}$ to the problem of fine-tuning LLMs from multi-objective LLM-as-a-Judge feedback (e.g., rubric-based judges), a setting where both sources of intransitivity arise. We find that $\texttt{PROSPER}$ outperforms all baselines considered across both instruction following and general chat benchmarks, releasing trained model checkpoints at the 7B and 3B parameter scales.

• The paper proposes the MaxEntBW solution, which robustly optimizes LLM fine-tuning under intransitive multi-objective feedback by reformulating the problem as a single-player optimization.
• It introduces the PROSPER algorithm, leveraging regression-based mirror descent and entropy regularization to achieve scalable, closed-form adversary optimization.
• Empirical results demonstrate that PROSPER-trained models outperform baselines by approximately 2/3 in LLM judge win-rates, validating its efficacy in realistic multi-criteria settings.

Addressing Intransitivity in Multi-Objective Preference Fine-Tuning: The MaxEnt Blackwell Winner and PROSPER Algorithm

Introduction

Preference fine-tuning (PFT) of LLMs increasingly leverages feedback from LLM "judges" that assess responses along multiple, often prompt-dependent objectives. A central challenge that arises in this setting is the ubiquity of intransitive preferences—preference cycles—either due to inconsistency within single objectives or the scalarization of multiple criteria. Traditional PFT pipelines, typically predicated on the existence of a total ordering via transitive preferences, are ill-equipped for this reality. "Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning" (2602.19041) provides a new game-theoretic framework for robust PFT under intransitive, multi-objective preferences, introducing the Maximum Entropy Blackwell Winner (MaxEntBW) and a scalable, regression-based algorithm for its realization, PROSPER.

Figure 1: Schematic of learning from multiple, possibly intransitive, LLM judge preferences and overview of the MaxEntBW framework and PROSPER algorithm.

Problem Formulation and Conceptual Framework

The paper formalizes the empirical setting where, for each prompt $x$, a multi-objective pairwise judge $P$ compares outputs over $m(x)$ criteria, producing preference signals that can be cyclic both within and across objectives. Under intransitivity, core social choice theory results—such as the existence of a Condorcet Winner or applicability of the Minimax Theorem—are invalidated, thus undermining the foundations of standard game-based or reward model-based preference optimization.

The authors generalize prior solution concepts—namely Maximal Lotteries/von Neumann Winners (for single-objective intransitivity) and the Blackwell Winner (for multi-objective settings)—by proposing the Maximum Entropy Blackwell Winner (MaxEntBW). Here, robustness is defined in terms of being locally maximally preferred against all policies, across all criteria, with additional adversarial regularization via a KL penalty to control the strength of the competitor policy.

The Maximum Entropy Blackwell Winner: Theory and Reduction

The MaxEntBW is defined as the policy maximizing

$$V(\pi) = \min_{w(x),\, \pi' \in \Pi} \mathbb{E}_x\left[\langle w(x), P(\pi \succ \pi'|x)\rangle + \beta D(\pi'(x)\|\pi_{ref}(x))\right]$$

where $w(x)$ is a distribution over criteria and the KL penalty with coefficient $\beta$ regularizes the adversary. This objective sidesteps the need for the Minimax Theorem by reformulating the problem in terms of a single-player optimization over the learner’s policy, eliminating adversarial training between two policies, and reducing the inner optimization to a closed-form via entropy regularization.

Key theoretical reductions include:

• Entropy regularization yields a tractable, closed-form adversary.
• Vertex elimination of the $w(x)$ optimization, leveraging concavity, enables focusing on per-criterion minimization.
• Online convex optimization with regression-based mirror descent translates the remaining policy optimization into scalable square-loss regression, making implementation tractable even for large models/prompt spaces.

Theoretical guarantees are established: under standard regression approximation assumptions, PROSPER efficiently approximates the MaxEntBW up to known concentrability and regression errors.

PROSPER: Scalable Multi-Objective Preference Optimization

The PROSPER algorithm (PReference Optimization with a Single Player over Entire Rubrics) operationalizes the MaxEntBW solution via regression-based mirror descent. The training loop involves generating candidate/pairwise responses, drawing sampled evaluations from reference and current policies, and regressing estimated policy gradients derived from the closed-form entropy-regularized adversarial objectives. The regression target captures the difference in gradients between the current and reference policy distributions, with per-prompt and per-criteria minimums selected as adversarially as possible but within the computational tractability imposed by the closed-form reductions.

Empirical results are supported by rigorous experimental protocols, including the use of prompt-specific rubrics (WildChecklists), recent LLM architectures (Qwen2.5, Qwen3), and multiple judge models.

Empirical Analysis: Intransitivity and Multi-Objective Feedback

Empirical investigation reveals that intransitive preferences are widespread in current LLM-judge feedback. Even when criteria are explicitly separated (thus removing the confound of scalarization-based intransitivity), preference inconsistencies persist at significant rates—only mitigated, not eliminated, by decomposition.

Figure 2: Proportion of prompts with no Condorcet Winner (left) and overall intransitive preferences (right) for joint-criterion and separated-criterion judges as response set size increases. Decomposition of criteria reduces but does not remove intransitivity.

This validates the central claim that algorithms must robustly optimize in settings with recurrent multi-objective intransitivity; simple rubric decomposition is insufficient.

Comparative Performance: Benchmarks and Win-Rates

The trained policies via PROSPER (with released 3B/7B models) are evaluated against RLCF (RL from Checklist Feedback) and multiple ablated baselines (single-objective, fixed comparator). Benchmarks include instruction following (AlpacaEval), general chat (Arena-Hard), and diverse QA and reasoning datasets (IFEval, MMLU, etc.). Strong claims are substantiated:

• PROSPER policies consistently outperform RLCF (by ~2/3), single-objective and fixed-competitor ablations, and the base model in LLM judge win-rates on held-out prompts.
• On instruction following and chat benchmarks, PROSPER yields best or second-best performance across evaluation modes, with no significant regression on out-of-domain benchmarks (demonstrating maintenance of core capabilities).

  Figure 3: PROSPER-trained policies outperform RLCF, the base model, and all ablations in average LLM judge win-rate on held-out prompts, showing superior multi-criteria optimization.

Policy Evaluation and LLM Judge Prompts

Prompting methodology for LLM-judge scoring is made explicit (Figures 4 and 5), leveraging both single-criterion (for decomposed feedback) and multi-criterion (for aggregate feedback) templates, and addressing positional bias and score normalization.

Figure 4: Prompt template for generating preference scores according to a single criterion per item.

Figure 5: Prompt template for joint, multi-criteria preference assessment.

Implications and Future Work

This work delivers several key theoretical and practical advances:

• Well-posed optimization framework for PFT under ubiquitous intransitive preferences over multiple objectives.
• Algorithmic reduction that removes the need for inefficacious adversarial training and excessive scalarization.
• Demonstration of scalable, numerically superior fine-tuning for LLMs in realistic, multi-criteria judge feedback settings.

This closes an important gap between social choice theory’s negative results (on ranking from intransitive aggregation) and practical LLM fine-tuning. Future work will likely extend these solution concepts to richer forms of feedback, beyond pairwise comparisons, to federated or interactive judge population settings, and to theoretical robustness analyses in open-ended preference spaces. The MaxEntBW and PROSPER framework constitutes a robust foundation for preference-based RL from LLM (and potentially human) feedback under multi-objective, non-transitive scenarios.

Conclusion

The MaxEntBW solution concept and the PROSPER algorithm mark a substantive progression in the robust optimization of LLM policies from real-world, multi-objective preference feedback that is inherently intransitive and rubric-driven. Comprehensive empirical results, strong theoretical underpinnings, and reproducible model releases collectively position this approach as a reference architecture for future PFT under complex judge behaviors.