Consensus Preference Optimization Methods

• Consensus Preference Optimization is a family of methods that derive an aggregate consensus from diverse individual preferences to optimize decisions, rankings, or model parameters.
• These methods utilize probabilistic models such as Bayesian inference, latent Gaussian processes, and vote-aware aggregation to reconcile heterogeneous signals.
• The approaches maintain minority views while improving predictive performance in applications like recommendation systems, group decision-making, and machine learning alignment.

Consensus Preference Optimization denotes a family of methods that optimize decisions, rankings, or model parameters against an aggregate preference signal derived from multiple annotators, users, agents, or objectives. Across the literature, the term is used in several technically distinct ways: as Bayesian inference of a crowd consensus preference function from pairwise labels, as vote-aware preference optimization for LLMs, as optimization of agreement or promotion objectives in group decision-making, as social-influence-free Bayesian optimization over collective utilities, and as consensus formation in distributed or multi-objective optimization (Simpson et al., 2019, Cho et al., 2024, Adachi et al., 11 Feb 2025, Jiang, 20 May 2026). In all of these formulations, the central problem is not merely to collect preferences, but to define what counts as “consensus,” quantify uncertainty or disagreement around it, and optimize toward it without collapsing heterogeneous preferences into an unexamined majority label.

1. Core formulations of consensus

A first major formulation treats consensus as a latent population-level preference object. In scalable Bayesian crowd preference learning, the goal is to learn both individual preference functions $f_u(i)$ and a crowd consensus preference function over items from sparse pairwise labels $y_{u,ij}\in\{0,1\}$. The consensus is a shared function $t(x_i)$ over item features, while personal preferences are represented as user-specific deviations around that shared structure (Simpson et al., 2019). A closely related personalized attribute-learning formulation decomposes each annotator model as $W^{(i)}=\theta+G^{(i)}+P^{(i)}$, where $\theta$ is an explicit consensus component, $G^{(i)}$ captures group-level deviations, and $P^{(i)}$ captures annotator-specific idiosyncrasies (Yang et al., 2019).

A second formulation treats consensus as the degree of agreement in observed votes. In vote-based preference optimization for LLM alignment, the preference target is not a hard binary label but a calibrated posterior mean $\hat p$ derived from vote counts $(n_+,n_-)$, so that “obvious” pairs and “controversial” pairs contribute differently to training (Cho et al., 2024). In self-improving code generation, consensus is likewise operationalized as behavioral agreement across candidate programs: a Direct Preference Optimization variant weights each preference pair by the consensus score of the winning program’s execution behavior over test inputs (Zhang et al., 31 Mar 2026).

A third formulation defines consensus through group decision objectives. In discrete choice models of group decisions, agreement is formalized as minimizing a disagreement functional

$$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$

where $y_{u,ij}\in\{0,1\}$0 is an intervention on the choice set (Tomlinson et al., 2020). In social Bayesian optimization, the consensus target is the social-influence-free decision

$y_{u,ij}\in\{0,1\}$1

with $y_{u,ij}\in\{0,1\}$2 a social welfare operator over individual utilities (Adachi et al., 11 Feb 2025).

A fourth formulation treats consensus as agreement among models or objectives during optimization. In decentralized DPO, consensus is parameter agreement across nodes, quantified by the network disagreement

$y_{u,ij}\in\{0,1\}$3

with consensus achieved when $y_{u,ij}\in\{0,1\}$4 (Jiang, 20 May 2026). In Pareto-Lenient Consensus, consensus is coalition-level agreement among multiple preference objectives, summarized by a surplus surrogate $y_{u,ij}\in\{0,1\}$5 and culminating in a Pareto consensus equilibrium rather than a fixed scalarized compromise (Tan et al., 7 Apr 2026).

2. Bayesian crowd consensus from pairwise comparisons

The most explicit probabilistic treatment of consensus preference optimization in the provided literature is the CrowdGPPL framework. The setting consists of $y_{u,ij}\in\{0,1\}$6 users, $y_{u,ij}\in\{0,1\}$7 items, item features $y_{u,ij}\in\{0,1\}$8, user features $y_{u,ij}\in\{0,1\}$9, and pairwise labels $t(x_i)$0 indicating whether user $t(x_i)$1 prefers item $t(x_i)$2 over item $t(x_i)$3. The model adopts the Thurstone–Mosteller case-V probit likelihood

$t(x_i)$4

after setting $t(x_i)$5, and uses a low-rank factorization with a shared consensus term,

$t(x_i)$6

Here $t(x_i)$7 is the consensus GP, while $t(x_i)$8 and $t(x_i)$9 encode item-side and user-side latent structure (Simpson et al., 2019).

This construction makes consensus an explicit latent function rather than a post hoc aggregation rule. The posterior mean of the consensus score is

$W^{(i)}=\theta+G^{(i)}+P^{(i)}$0

and consensus rankings are obtained by sorting items by $W^{(i)}=\theta+G^{(i)}+P^{(i)}$1. The same model yields calibrated pairwise probabilities through Gaussian posterior marginalization, so uncertainty in the latent utilities is propagated into prediction rather than discarded. Because item and user features enter through kernels $W^{(i)}=\theta+G^{(i)}+P^{(i)}$2 and $W^{(i)}=\theta+G^{(i)}+P^{(i)}$3, the framework also supports cold-start prediction for new items and new users (Simpson et al., 2019).

Scalability is handled by sparse Gaussian processes with inducing points and stochastic variational inference. The non-conjugate probit likelihood is approximated by a moment-matched Gaussian likelihood plus a first-order Taylor expansion, and mini-batch updates are used for the inducing variables of the item-side, user-side, and consensus GPs. For single-user GPPL, per-iteration complexity is $W^{(i)}=\theta+G^{(i)}+P^{(i)}$4 in time and $W^{(i)}=\theta+G^{(i)}+P^{(i)}$5 in memory, with costs independent of dataset size $W^{(i)}=\theta+G^{(i)}+P^{(i)}$6 when $W^{(i)}=\theta+G^{(i)}+P^{(i)}$7 and $W^{(i)}=\theta+G^{(i)}+P^{(i)}$8 are fixed; crowdGPPL generalizes this with separate inducing sets for items and users (Simpson et al., 2019).

Empirically, the framework was evaluated on simulations, the Sushi recommendation datasets, and UKPConvArgCrowdSample. On UKPConvArgCrowdSample, consensus performance improved over GPPL and crowdBT-GP, with consensus Acc $W^{(i)}=\theta+G^{(i)}+P^{(i)}$9 versus $\theta$0 for GPPL and Kendall’s $\theta$1 $\theta$2 versus $\theta$3; on personal prediction for all workers, crowdGPPL achieved Acc $\theta$4 and $\theta$5 $\theta$6 versus GPPL $\theta$7 (Simpson et al., 2019). On Sushi-A, crowdGPPL achieved Acc up to $\theta$8 and $\theta$9 up to $G^{(i)}$0. The paper also reports that removing the consensus term $G^{(i)}$1 degrades performance, indicating that consensus is not merely an interpretive artifact but a functional component of the predictive model (Simpson et al., 2019).

A related but distinct line of work models consensus-to-personal evolution in annotator ranking through multi-task AUC optimization. There, the consensus parameter $G^{(i)}$2 is regularized jointly with a low-rank group matrix $G^{(i)}$3 and a sparse personal matrix $G^{(i)}$4, and the objective is an AUC surrogate written as a Laplacian quadratic form. This framework provides a closed-form proximal step for the non-convex group subproblem, convergence to critical points, and a generalization bound, while preserving an explicit separation between mass opinion and personalized deviations (Yang et al., 2019).

3. Vote-aware and consensus-weighted alignment objectives

In LLM alignment, one influential use of consensus preference optimization is to retain the number of votes rather than collapsing multiple annotations into a binary winner. Vote-based Preference Optimization models the unknown probability $G^{(i)}$5 that $G^{(i)}$6 is preferred over $G^{(i)}$7 with a symmetric Beta prior $G^{(i)}$8, yielding the Bayesian MMSE estimator

$G^{(i)}$9

This $P^{(i)}$0 replaces hard labels in preference optimization objectives, so that learning strength depends on consensus intensity rather than only on preference direction (Cho et al., 2024).

The framework extends standard DPO and IPO into VDPO and VIPO. VDPO uses both directions of the comparison with vote-informed target probabilities: $P^{(i)}$1 while VIPO converts $P^{(i)}$2 into a target margin

$P^{(i)}$3

This makes controversial pairs behave like adaptive label smoothing and obvious pairs induce stronger preference margins. The paper reports that on SHP and UFB, VDPO and VIPO consistently outperform their base algorithms; on UFB with Pythia 2.8B, VDPO achieved $P^{(i)}$4 in-domain versus DPO’s $P^{(i)}$5, and the analysis reports that VDPO reduces reward divergence while DPO’s implicit reward margin grows unbounded without early stopping (Cho et al., 2024).

A different consensus-balanced alignment strategy appears in Steerable Cultural Preference Optimization of reward models. There, consensus is operationalized through a global reward model $P^{(i)}$6, used as a proxy for mainstream or consensus preference. Minority-country preference pairs are filtered by the global Bradley–Terry probability

$P^{(i)}$7

retaining pairs with $P^{(i)}$8, and then weighted by

$P^{(i)}$9

The resulting objective down-weights strongly divergent examples while preserving culturally specific signals (Oh et al., 17 Jun 2026). On PRISM, minority reward model performance improved by up to 7 points over the baseline across two datasets and across 7 countries, and SCPO was reported as up to 280% more training data-efficient than full-data finetuning (Oh et al., 17 Jun 2026).

Consensus-weighted DPO also appears in self-improving code generation. Con-DPO defines a behavioral consensus score for a candidate program $\hat p$0 from execution outputs across test inputs: $\hat p$1 with crashed candidates assigned $\hat p$2. Preference pairs are then weighted by the winner’s consensus,

$\hat p$3

This design is intended to suppress noisy self-generated supervision when no teacher and no reliable test oracle are available (Zhang et al., 31 Mar 2026). On three code backbones, ConSelf with Con-DPO improved over base models, for example raising Qwen2.5-Coder-7B from $\hat p$4 to $\hat p$5 average pass@1, while ablations showed further degradation when either curriculum curation or consensus weighting was removed (Zhang et al., 31 Mar 2026).

4. Consensus at inference time and as distilled decoding policy

Another branch of the literature treats consensus preference optimization as an inference-time or decoding-time procedure, sometimes later amortized into training. Robust Preference Selection models a user preference as a direction $\hat p$6, defines a local neighborhood $\hat p$7, samples candidates under nearby preferences $\hat p$8, and then selects the candidate maximizing the original-intent score

$\hat p$9

The consensus here is local and directional: it is a consensus over nearby trade-off vectors rather than over annotator votes. Under first-order stochastic dominance of neighborhood samples over single-direction samples, the paper proves

$(n_+,n_-)$0

Empirically, under strict compute parity with best-of-$(n_+,n_-)$1, RPS improved robustness across DPA, DPO, and SFT models, with mean win rates such as $(n_+,n_-)$2 on UltraFeedback for DPA and gains increasing with preference angle; for DPA on UltraFeedback, win rate rose from $(n_+,n_-)$3 at $(n_+,n_-)$4 to $(n_+,n_-)$5 at $(n_+,n_-)$6 (Mao et al., 23 Oct 2025).

Consensus Group Relative Policy Optimization internalizes a different inference-time consensus rule: Minimum Bayes Risk decoding. For a prompt $(n_+,n_-)$7 and utility $(n_+,n_-)$8, MBR uses the expected utility

$(n_+,n_-)$9

or its Monte Carlo estimate

$$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$0

and selects the hypothesis with maximal expected utility. C-GRPO defines the GRPO reward as this self-consensus utility over the sampled group, so the training signal depends only on policy samples and the task utility, not on gold references or explicit preference labels (Ichihara et al., 3 Feb 2026).

The theoretical result is a directional-alignment theorem: under the paper’s assumptions, the expected GRPO gradient is proportional to the gradient of the expected-utility objective underlying MBR,

$$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$1

for some $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$2, and stochastic gradient ascent yields the non-asymptotic bound

$$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$3

The practical purpose is amortization: the $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$4 pairwise scoring cost of MBR is paid during training, while inference becomes single-pass generation (Ichihara et al., 3 Feb 2026). On WMT 2024 machine translation, average COMET scores were $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$5 for C-GRPO and $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$6 for C-Dr.GRPO, compared with $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$7 for MBR decoding, and on XSum with Llama the ROUGE-Lsum score increased from $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$8 for the base model to $$D(Z)=\sum_{\{a,b\}\subseteq \mathcal{U}}\sum_{x\in C}\left|P_a(x\mid C\cup Z)-P_b(x\mid C\cup Z)\right|,$$9 for C-GRPO (Ichihara et al., 3 Feb 2026).

These two approaches illustrate complementary meanings of consensus at inference. In RPS, consensus is a local neighborhood agreement used at test time without retraining. In C-GRPO, consensus is a sample-and-rerank utility distilled into policy parameters so that the test-time consensus computation can be removed.

5. Group decision, social choice, and choice-environment design

In social choice and decision analysis, consensus preference optimization often concerns how a group ranking or group-optimal decision should be constructed from heterogeneous preferences. Convergence Voting transforms the Condorcet graph of pairwise comparisons into a Markov chain with transition matrix $y_{u,ij}\in\{0,1\}$00, and uses its stationary distribution $y_{u,ij}\in\{0,1\}$01 as a consensus score vector. The stationary allocation is interpreted as “negotiated community support,” obtained by iterative reallocation of support along pairwise-preference directions until convergence (Bana et al., 2021). The method is positioned as a consensus voting rule between Copeland and Borda: it need not choose the Condorcet winner, but it reduces the influence of unpopular candidates relative to Borda without introducing ad hoc weighting (Bana et al., 2021).

A fuzzy-preference variant formalizes consensus jointly with internal rationality. In group decision making with fuzzy preference relations $y_{u,ij}\in\{0,1\}$02, additive transitivity

$y_{u,ij}\in\{0,1\}$03

defines consistency, while pairwise similarities across experts define the consensus relation. The paper combines global consistency and global consensus into

$y_{u,ij}\in\{0,1\}$04

and uses simulated annealing to optimize this consistency/consensus level by modifying experts’ preference structures (Das et al., 2014). In the reported experiment with $y_{u,ij}\in\{0,1\}$05, optimization moved the system from $y_{u,ij}\in\{0,1\}$06, $y_{u,ij}\in\{0,1\}$07, $y_{u,ij}\in\{0,1\}$08 to $y_{u,ij}\in\{0,1\}$09, $y_{u,ij}\in\{0,1\}$10, $y_{u,ij}\in\{0,1\}$11, illustrating an explicit trade-off in which consensus increases at some cost to consistency (Das et al., 2014).

A more interventionist interpretation appears in discrete choice modeling. Here the problem is not merely to aggregate existing opinions, but to optimize the presented choice set $y_{u,ij}\in\{0,1\}$12 so as to maximize agreement, maximize disagreement, or promote a target choice. Agreement is defined by minimizing the disagreement functional $y_{u,ij}\in\{0,1\}$13, but the paper proves that Agreement is NP-hard in MNL, and therefore in CDM, NL, and EBA through model embeddings; Disagreement is likewise NP-hard, while Promotion is NP-hard in CDM, NL, and EBA but can become tractable under specific restrictions such as same-tree NL or disjoint-aspect EBA (Tomlinson et al., 2020). The work also provides an $y_{u,ij}\in\{0,1\}$14-additive approximation algorithm for Agreement in MNL with runtime

$y_{u,ij}\in\{0,1\}$15

and exact mixed-integer bilinear formulations for moderate-size instances (Tomlinson et al., 2020).

In engineering-oriented group decision making, ODESYS/FIVES pushes the term further toward design optimization. The framework requires Preference-Key, Integration, Association, and Uniqueness, maps all performances into a unified preference domain via actor-specific preference functions $y_{u,ij}\in\{0,1\}$16, and maximizes an affine aggregate

$y_{u,ij}\in\{0,1\}$17

Its claim is that group decision validity requires aggregation in preference space rather than directly over heterogeneous objectives, and that the solver should return a single best-fit-for-common-purpose solution rather than a Pareto set (Wolfert, 19 Mar 2026). This suggests a broader usage of consensus preference optimization in which consensus is the unique maximizer of an affine preference aggregate subject to feasibility and acceptability constraints.

6. Social influence, decentralization, and multi-preference negotiation

Social Bayesian Optimization defines consensus as the maximizer of a social-influence-free welfare function. Each agent $y_{u,ij}\in\{0,1\}$18 has a black-box utility $y_{u,ij}\in\{0,1\}$19, the social utility is $y_{u,ij}\in\{0,1\}$20, and public votes are influenced by a row-stochastic social graph $y_{u,ij}\in\{0,1\}$21 through

$y_{u,ij}\in\{0,1\}$22

Private votes follow the true utilities $y_{u,ij}\in\{0,1\}$23, public votes follow the influenced utilities $y_{u,ij}\in\{0,1\}$24, and both are modeled via Bradley–Terry pairwise feedback (Adachi et al., 11 Feb 2025). A central theoretical result is that, under positive-linear aggregation and non-trivial consensus, no aggregation rule is groupthink-proof: using public votes alone cannot in general recover the social-influence-free consensus. SBO therefore uses dual voting, estimates the social graph faster than the utilities themselves, debiases public votes via $y_{u,ij}\in\{0,1\}$25 when feasible or via the constraint $y_{u,ij}\in\{0,1\}$26 otherwise, and optimizes an acquisition function

$y_{u,ij}\in\{0,1\}$27

The method achieves sublinear private-vote complexity and was evaluated on thermal comfort, team building, travel negotiation, and energy trading collaboration (Adachi et al., 11 Feb 2025).

Distributed Direct Preference Optimization studies consensus when preference data are fragmented across users or devices. In the decentralized regime, clients exchange parameters through a symmetric, doubly-stochastic mixing matrix, and the spectral quantity

$y_{u,ij}\in\{0,1\}$28

governs mixing speed. The main convergence theorem yields

$y_{u,ij}\in\{0,1\}$29

Hence optimization speed and consensus quality are explicitly controlled by the spectral gap $y_{u,ij}\in\{0,1\}$30 and by preference heterogeneity $y_{u,ij}\in\{0,1\}$31 (Jiang, 20 May 2026). Experiments on SHP and Anthropic HH-RLHF corroborate the theoretical dependence on participation, local steps, staleness, and topology.

Pareto-Lenient Consensus generalizes the notion of consensus from users to objectives. In multi-preference RLHF, the method partitions per-token advantages into a coherent coalition $y_{u,ij}\in\{0,1\}$32 and a conflict set $y_{u,ij}\in\{0,1\}$33, defines the coalition surplus surrogate

$y_{u,ij}\in\{0,1\}$34

and applies the lenient mask

$y_{u,ij}\in\{0,1\}$35

The rectified update is then

$y_{u,ij}\in\{0,1\}$36

The stated goal is to escape risk-averse equilibria where strict conflict avoidance traps optimization in conservative local compromises, and to converge instead to a Pareto consensus equilibrium characterized by negligible coalition surplus (Tan et al., 7 Apr 2026). On BeaverTails, PLC improved hypervolume by about 31.7% over RiC, and in the tri-objective setting it achieved about 35% hypervolume gain over GAPO (Tan et al., 7 Apr 2026).

7. Common themes, assumptions, and limitations

Across these formulations, consensus is typically not reduced to a raw majority vote. It may be a latent GP function $y_{u,ij}\in\{0,1\}$37, a Bayesian posterior mean $y_{u,ij}\in\{0,1\}$38, a stationary distribution $y_{u,ij}\in\{0,1\}$39, a social welfare maximizer $y_{u,ij}\in\{0,1\}$40, a group-relative expected utility, a network agreement condition, or a coalition-level equilibrium (Simpson et al., 2019, Cho et al., 2024, Bana et al., 2021, Adachi et al., 11 Feb 2025, Ichihara et al., 3 Feb 2026, Tan et al., 7 Apr 2026). This diversity means that the phrase “Consensus Preference Optimization” is context-dependent rather than canonically standardized.

A second recurring feature is that consensus and heterogeneity are usually modeled jointly rather than treated as mutually exclusive. CrowdGPPL combines the shared function $y_{u,ij}\in\{0,1\}$41 with user-specific latent factors; multi-task AUC optimization separates consensus, group, and personal components; SCPO preserves minority reward modeling while regularizing it against a global reference; distributed DPO quantifies client drift; and PLC explicitly negotiates among conflicting objectives rather than eliminating them (Simpson et al., 2019, Yang et al., 2019, Oh et al., 17 Jun 2026, Jiang, 20 May 2026, Tan et al., 7 Apr 2026). A plausible implication is that modern uses of the term increasingly view consensus as structured aggregation under disagreement, not as homogenization.

The literature also shares strong modeling assumptions. Vote-aware methods typically assume i.i.d. Bernoulli votes with symmetric Beta priors; crowd preference learning often assumes independent pairwise labels conditioned on latent utilities and uses Thurstone–Mosteller or Bradley–Terry likelihoods; social-influence models assume a linear graph convolution $y_{u,ij}\in\{0,1\}$42; distributed analyses rely on smoothness, bounded variance, and spectral conditions; code self-improvement assumes that executional agreement over available test inputs is an informative proxy for correctness (Cho et al., 2024, Simpson et al., 2019, Adachi et al., 11 Feb 2025, Jiang, 20 May 2026, Zhang et al., 31 Mar 2026). When these assumptions fail, the resulting “consensus” can become biased, unstable, or weakly identified.

Finally, several works emphasize computational or theoretical limits. Agreement optimization under standard discrete choice models is NP-hard; public-vote-only recovery of social-influence-free consensus is impossible outside trivial cases; scalable Bayesian crowd models rely on likelihood approximations and fixed kernels; and consensus-weighted self-training can still fail when the model is far beyond its capability or when proxy utilities are misaligned (Tomlinson et al., 2020, Adachi et al., 11 Feb 2025, Simpson et al., 2019, Zhang et al., 31 Mar 2026). These results distinguish consensus preference optimization from simple label aggregation: it is a modeling and optimization problem whose difficulty depends on what is being aggregated, how disagreement is represented, and what guarantees are sought.