Majority Voting Reward Function

• Majority Voting Reward Function is a method that aggregates individual votes or model outputs based on frequency counts to assign binary rewards.
• It addresses issues like sparse supervision and confirmation bias, while facing challenges such as minority-correct outputs and the pit of losses in stochastic environments.
• Advanced variants, including confidence-weighted and subgroup consensus methods, enhance reward accuracy in reinforcement learning and collective decision-making.

A majority voting reward function is a decision-theoretic or learning-theoretic construct in which collective outcomes, pseudo-labels, or direct rewards are determined by aggregating multiple votes—each corresponding to an agent’s recommendation, model output, or sampled solution—via majority rule. This paradigm underlies core methods in reinforcement learning with pseudo-labels, collective decision theory, and self-supervised reward shaping, and is central to current test-time reinforcement learning for LLMs, stochastic voting in societal environments, and mechanism design for truthful aggregation. Both static and dynamically-weighted variants exist, as well as extensions overcoming canonical limitations of naïve majority aggregation.

1. Mathematical Implementation of the Majority Voting Reward

The classic majority voting reward is defined by aggregating a set of outputs or votes $\{o_i\}_{i=1}^N$ for a given input using frequency or confidence counts. For each candidate answer $y\in\mathcal{Y}$, the vote count is

$$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$

The majority-voted pseudo-label is then

$y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$

For each output $o_i$, the reward is assigned as

$$r_\mathrm{MV}(o_i) = \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y^*_\mathrm{MV}\bigr].$$

This reward function is widely used in pseudo-labeling weakly supervised learning and test-time reinforcement learning, including language modeling and complex problem solving (Wang et al., 17 Dec 2025, Zhang et al., 1 Aug 2025).

2. Limitations and Pathologies of Majority Voting Rewards

Several works have identified inherent limitations of the majority voting reward:

• Sparse Supervision and Confirmation Bias: Majority-based rewards provide a single binary feedback per datapoint, often failing to reward minority-but-correct outputs and amplifying confirmation bias in overrepresented, low-quality solutions (Wang et al., 17 Dec 2025, Zhang et al., 1 Aug 2025).
• The Pit of Losses Paradox in Stochastic Environments: Within the ViSE (Voting in Stochastic Environment) model, majority voting can destroy capital in both hostile ($\mu<0$) and highly favorable ($\mu>0$) environments, yielding expected one-step rewards less than those from trivially rejecting or accepting all proposals. This is expressed as

$$E_X[\Delta_i] < 0 \quad \text{for } \mu/\sigma < -0.25 \quad (\text{Gaussian } X)$$

and analogously in highly favorable regimes (Chebotarev et al., 2023).

• Majority Is Not Always Correct: Empirically, hard aggregation tasks frequently exhibit cases where the true answer is a minority or even singleton solution, especially as the complexity of the reasoning problem increases (Zhao et al., 8 Sep 2025).

These issues motivated the development of refined and alternative reward functions.

3. Advanced Extensions: Confidence-Weighted and Subgroup Rewards

To address these weaknesses, several designs extend the majority voting reward:

• Stepwise Confidence-Weighted Voting: Decomposes outputs into reasoning steps and assigns token-level confidences. The confidence-weighted majority pseudo-label is

$$y^*_\mathrm{CW} = \arg\max_{y\in\mathcal{Y}} \sum_{i=1}^N C_i\,\mathbf{1}[\mathrm{Ans}(o_i)=y],$$

where $y\in\mathcal{Y}$0 measures average step confidence per output (Wang et al., 17 Dec 2025).

• Local Subgroup Consensus (SCOPE Framework): The output pool is dynamically partitioned into subgroups $y\in\mathcal{Y}$1, and local weighted majorities $y\in\mathcal{Y}$2 are computed per subgroup. The reward is

$y\in\mathcal{Y}$3

and subgroup size is chosen via Pareto optimization to balance local consensus strength and answer diversity. This produces multiple supervision signals per example and mitigates signal sparsity (Wang et al., 17 Dec 2025).

A summary of these variants:

Reward Variant	Formula	Key Feature
Majority-Voting ($y\in\mathcal{Y}$4)	$y\in\mathcal{Y}$5	Binary per global majority
Confidence-Weighted ($y\in\mathcal{Y}$6)	$y\in\mathcal{Y}$7	Incorporates stepwise confidence
Subgroup SCOPE ($y\in\mathcal{Y}$8)	$y\in\mathcal{Y}$9	Dense, multi-label supervision

4. Collective Decision-Theoretic Formulations

In stochastic collective choice—exemplified by the ViSE model—rewards from majority voting are linked to societal capital increments. Each proposal is an $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$0-vector $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$1 of i.i.d. gains, and symmetrized majority approves proposals with $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$2 yes votes, with tie-breaking at $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$3 using probability $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$4:

$$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$5

The expected reward for agent $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$6 is

$$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$7

where $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$8 and $$V(y) = \sum_{i=1}^N \mathbf{1}\bigl[\mathrm{Ans}(o_i) = y\bigr].$$9 (Chebotarev et al., 2023).

A notable identity is the mirror-symmetry:

$y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$0

for all $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$1 and $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$2 with mean $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$3. This underlies the mirrored performance and the emergence of "twin pits of losses"—regimes on both sides of neutrality where majority rewards are strictly inferior to always rejecting or always accepting proposals.

5. Mechanism Design and Strategic Incentives

Mechanism-design approaches implement majority rule and associated rewards via structured games:

• Bloc-Formation Mechanism: Agents submit a vote and a cooperation set; outcomes are determined by majority blocs. Off-equilibrium, a lottery $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$4 is enforced, assigning outcome probabilities based on coalition nominations. Truthful bloc formation yields Nash equilibrium, as any deviation decreases a voter's expected reward (Margarita et al., 2023).
• Random-Confirmations Mechanism: Voting is followed by random sampling of confirmers. If no confirmers approve, a lottery $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$5, proportional to vote shares, is executed. Equilibrium is reached with sincere majority voting and confirmation. This maintains subgame perfect implementation of the majority decision (Margarita et al., 2023).

In both cases, the "reward" is the probabilistic outcome—either deterministic (upon majority) or randomized (in off-equilibrium scenarios)—and is strictly increasing in an agent’s truthful support for her favorite alternative.

6. Majority Voting Reward in Reinforcement Learning and LLMs

Majority voting reward functions are fundamental in LLM RL:

• Self-Consistency and Pseudo-Labeling: In test-time RL, sampling $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$6 rollouts, aggregating answers, and using the majority as a reward enables unsupervised or weakly supervised policy improvement. For rollout $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$7, reward is $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$8 (Zhang et al., 1 Aug 2025).
• Contrastive and Cross-Referencing Rewards (Co-Reward): By constructing semantically analogous prompts, computing majority pseudo-labels on each, and cross-referencing rewards across analogues, one enforces consistency and improves reward robustness, mitigating reward collapse (Zhang et al., 1 Aug 2025).
• Aggregator Learning (AggLM): Instead of static voting, an aggregator model is trained to select or synthesize the final answer, rewarded only if it matches ground truth, enabling minority-correct answer recovery (Zhao et al., 8 Sep 2025).

7. Implications, Symmetries, and Tuning

The fundamental symmetry $y^*_\mathrm{MV} = \arg\max_{y\in\mathcal{Y}} V(y).$9 in majority voting reward functions demonstrates that adding environmental drift simply translates expected reward curves. As a consequence:

• In moderately neutral environments ($o_i$0), majority voting outperforms always-accept or always-reject strategies.
• For $o_i$1 above pit-thresholds, extremal deterministic rules (“accept all” or “reject all”) are superior.
• Tuning voting quotas, or introducing reward/penalty adjustments, can keep aggregation systems out of “pit of losses” regimes (Chebotarev et al., 2023).

Mechanism-design perspectives, confidence weighting, and subgroup partitioning are active research directions to improve the fidelity, density, and reliability of majority-based reward schemes in both artificial and collective-intelligence systems.

---

References:

• (Chebotarev et al., 2023) Majority voting is not good for heaven or hell, with mirrored performance
• (Wang et al., 17 Dec 2025) Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning
• (Zhao et al., 8 Sep 2025) The Majority is not always right: RL training for solution aggregation
• (Zhang et al., 1 Aug 2025) Co-Reward: Self-supervised Reinforcement Learning for LLM Reasoning via Contrastive Agreement
• (Margarita et al., 2023) Legitimacy of collective decisions: a mechanism design approach