Weighted Majority Algorithm Overview

• Weighted Majority Algorithm is an online learning method that uses multiplicative updates on expert weights to aggregate predictions and minimize mistakes.
• It supports various configurations including deterministic and randomized decision-making, and extends naturally to multi-class and loss-based settings.
• Its adaptations span distributed learning, ensemble classification, and PAC-Bayesian frameworks, enhancing both prediction accuracy and computational efficiency.

Searching arXiv for recent and foundational papers on the Weighted Majority Algorithm and related weighted majority voting. The Weighted Majority Algorithm (WMA) is an online learning procedure for prediction with expert advice in which a learner maintains nonnegative weights over a pool of experts, predicts by a weighted vote or by randomized selection according to normalized weights, and then updates those weights multiplicatively after the true outcome is revealed. In its classical binary mistake-based form, experts that err are penalized by a constant factor in $(0,1)$; in loss-based generalizations such as Hedge, the multiplicative update is exponentiated by the incurred loss. The same weighted-majority principle also appears as a broader weighted voting rule in ensemble classification, crowdsourcing, distributed online learning, and consensus systems, but these settings differ in their assumptions, update mechanisms, and guarantees (Ouyang et al., 2011, Georgiou et al., 2013).

1. Online expert-advice formulation

In the classical expert-advice setting, the learner faces rounds $t=1,\dots,T$ and has access to $P$ experts. At round $t$, expert $i$ predicts $y_{t,i}$, the environment reveals the outcome $l_t$, and the learner tracks an expert-specific loss $\ell_{t,i}$. The learner maintains weights $w_{t,i}\ge 0$ and predicts either by a weighted majority vote or by sampling from the normalized weights. For binary $0/1$ losses, the canonical mistake-based update leaves the weight unchanged when an expert is correct and multiplies it by $t=1,\dots,T$0 when the expert errs: $t=1,\dots,T$1 A standard deterministic decision rule predicts the label whose supporting experts have larger total weight (Ouyang et al., 2011).

This multiplicative-penalty structure extends naturally beyond binary voting. In a multi-class setting, a standard aggregation rule is

$t=1,\dots,T$2

This preserves the core weighted-majority mechanism while replacing a binary comparison by a classwise score maximization (Mukherjee et al., 2024).

A recurring terminological distinction is important. Several later literatures use “weighted majority voting” or “weighted majority rule” to denote the weighted decision rule itself, whereas the classical WMA denotes the online multiplicative-update algorithm built around that rule. In batch ensemble classification, for example, the weights may be computed analytically from estimated classifier competence rather than updated round by round (Georgiou et al., 2013).

2. Classical updates, randomized variants, and guarantees

For the mistake-based binary algorithm with unit initialization, a standard bound compares the learner’s total number of mistakes $t=1,\dots,T$3 with the number $t=1,\dots,T$4 made by the best expert in hindsight: $t=1,\dots,T$5 With base-2 logarithms and $t=1,\dots,T$6, this becomes

$t=1,\dots,T$7

where $t=1,\dots,T$8. The bound captures the central property of WMA: the learner’s error grows with the performance of the best expert plus a logarithmic dependence on the pool size (Ouyang et al., 2011).

A randomized version replaces the deterministic vote by sampling an expert proportionally to its weight. In the formulation used as the baseline for Cascading Randomized Weighted Majority, if expert $t=1,\dots,T$9 predicts incorrectly then its weight is multiplied by $P$0, and the expected number of learner mistakes satisfies

$P$1

where $P$2 is the number of mistakes made by the best expert so far. This variant is often presented as Randomized Weighted Majority (RWM) and serves as the immediate precursor to more structured specialist schemes (Zamani et al., 2014).

For general bounded losses $P$3, Hedge uses the exponentiated update

$P$4

followed by normalization. Its cumulative regret against any comparator expert $P$5 satisfies

$P$6

so choosing $P$7 yields $P$8 regret. In this sense, classical WMA, randomized weighted majority, and Hedge belong to the same multiplicative-weights family, differing mainly in the loss model and prediction rule (Ouyang et al., 2011).

3. Distributed and specialist extensions

A major extension of WMA treats online learning as a cooperative distributed process. In the data-distributed online learning framework, $P$9 agents learn synchronously on separate data streams, each performing a local online update and then a communication or mixing step over a connected graph. In the complete-graph case, this mixing is implemented by a doubly-stochastic matrix with entries $t$0, and the resulting Distributed Weighted Majority (DWM) algorithms instantiate mixing either by geometric or arithmetic averaging (Ouyang et al., 2011).

Distributed Weighted Majority by Imitation (DWM-I) applies the local multiplicative penalty with factor $t$1 and then mixes each expert weight geometrically: $t$2 Distributed Weighted Majority by Averaging (DWM-A) uses arithmetic mixing instead: $t$3 No further normalization is required because the weights remain nonnegative and predictions are made by weighted majority. For DWM-I, if $t$4 is the number of mistakes made by agent $t$5 and $t$6 is the total number of mistakes made so far by the best expert across all agents, then

$t$7

With $t$8, this gives

$t$9

The paper interprets this as roughly a $i$0 improvement when different agents observe complementary information; if all agents see identical data, the gain disappears because $i$1 scales with $i$2 (Ouyang et al., 2011).

The same work reports two operational consequences: with the same computation time, the distributed algorithms achieve smaller generalization errors than a single agent, and with the same generalization error they can be $i$3 times faster because work is parallelized. In experiments with $i$4 agents, simple decision-stump experts, and $i$5, each agent’s cumulative mistakes were reduced by roughly half relative to a single agent, and DWM-A consistently made slightly fewer mistakes than DWM-I (Ouyang et al., 2011).

A different structural extension is Cascading Randomized Weighted Majority (CRWM), which replaces the single global learner by a cascade of specialist RWM learners. In the binary construction, three learners are maintained: one specializing in negative predictions, one in positive predictions, and one fallback learner. Only the selected learner is updated on each round. The aggregate expected-mistake bound is

$i$6

where $i$7 is the number of mistakes made by the best expert of the $i$8-th learner so far. The paper shows that this bound is better than the RWM bound when the data size is large, provided at least one routed region has a specialist whose error rate is strictly better than that of the global best expert (Zamani et al., 2014).

4. Weighted majority as a statistical voting rule

Outside adversarial online learning, weighted majority also appears as a statistical decision rule with analytically chosen weights. In a game-theoretic formulation for binary ensemble classification, $i$9 classifiers cast hard votes $y_{t,i}$0, and the ensemble predicts

$y_{t,i}$1

Under conditional independence of experts given the true label, the analytically optimal weight is the log-odds of competence: $y_{t,i}$2 for global or local competence, respectively. This is a Bayes-optimal linear log-likelihood-ratio combiner under the stated assumptions (Georgiou et al., 2013).

The same framework estimates local competence $y_{t,i}$3 from classifier soft outputs using dynamic-width histograms and piecewise cubic Hermite spline interpolation. For each classifier, soft outputs on a validation set are binned so that each bin contains roughly the same number of samples; empirical binwise error rates are then interpolated by a shape-preserving spline $y_{t,i}$4, and local accuracy is defined as

$y_{t,i}$5

On Ringnorm, Splice, Twonorm, and Waveform, the adaptive weighted majority rule ranked first overall across SVM, OBTC, and weighted $y_{t,i}$6-NN ensembles, with accuracy improvements relative to the average group accuracy reaching up to approximately $y_{t,i}$7 in favorable cases (Georgiou et al., 2013).

When the source reliabilities used for weighting are estimated rather than known, two stability notions become relevant. For estimated trust $y_{t,i}$8 and random trustworthiness $y_{t,i}$9, stability of correctness holds: $l_t$0 provided $l_t$1 componentwise and the $l_t$2 are independent. Thus the believed accuracy equals the actual expected accuracy. Stability of optimality does not hold: decisions based on unbiased trust estimates can still be worse than those based on the true trustworthiness. Under bounded-support assumptions $l_t$3, the corresponding optimality gap satisfies

$l_t$4

and a weaker linear bound is also given. The paper further reports that overestimation of a highly trustworthy source tends to be more harmful than underestimation (Bai et al., 2022).

5. Second-order PAC-Bayesian theory for the weighted majority vote

A separate theoretical literature studies the generalization error of the weighted majority vote itself rather than the online multiplicative update. For a posterior distribution $l_t$5 on a hypothesis class $l_t$6, the multiclass weighted majority vote is

$l_t$7

The key second-order quantity is the tandem loss

$l_t$8

which measures joint error. A central oracle bound is

$l_t$9

and in binary classification this becomes

$\ell_{t,i}$0

where $\ell_{t,i}$1 is the disagreement probability. Because disagreement depends only on inputs, unlabeled data can be used to tighten estimation in the binary case (Masegosa et al., 2020).

The same work converts these oracle statements into PAC-Bayesian bounds that are explicitly optimizable over the voting distribution. In experiments on weighted random forests, minimizing the first-order bound concentrated nearly all weight on a few top trees and degraded test error, whereas minimizing the second-order bound distributed weight over many trees and typically did not degrade test error; in some cases it improved it. This suggests that the utility of weighted majority voting depends not only on average base-classifier accuracy but also on the correlation structure of their errors (Masegosa et al., 2020).

A further refinement uses a parametric form of the Chebyshev–Cantelli inequality. For any $\ell_{t,i}$2 and any $\ell_{t,i}$3,

$\ell_{t,i}$4

Applied to the multiclass weighted majority vote through the average loss $\ell_{t,i}$5, this yields

$\ell_{t,i}$6

Setting $\ell_{t,i}$7 recovers the second-order Markov or tandem bound, while optimizing $\ell_{t,i}$8 recovers the classical C-bound oracle form. The same paper derives a PAC-Bayes-Bennett inequality for empirical estimation and reports that the new bounds can improve on prior second-order oracle and empirical bounds (Wu et al., 2021).

6. Domain-specific adaptations and limitations

In crowdsourcing under the Dawid–Skene model, weighted majority voting becomes a label-aggregation rule rather than an online regret-minimization algorithm. For item $\ell_{t,i}$9 and class $w_{t,i}\ge 0$0, weighted majority voting uses

$w_{t,i}\ge 0$1

The Iterative Weighted Majority Voting (IWMV) method alternates between weighted aggregation and re-estimation of worker accuracies, updating weights by

$w_{t,i}\ge 0$2

Its one-step version has a provable theoretical guarantee in the binary homogeneous Dawid–Skene model, and the full IWMV procedure is reported to perform at least on par with state-of-the-art methods while having a much lower computational cost, around one hundred times faster than the state-of-the-art methods (Li et al., 2014).

In Proof-of-Stake consensus, weighted majority is used to aggregate validator approvals. Each validator has a voting profile $w_{t,i}\ge 0$3, updated multiplicatively according to whether its vote matches the eventual ground truth, and its decision weight is the log-odds

$w_{t,i}\ge 0$4

The accept rule is

$w_{t,i}\ge 0$5

with the welfare-optimal quota

$w_{t,i}\ge 0$6

Under the paper’s Bayesian welfare model, this rule maximizes expected collective welfare, and numerical experiments show improved robustness when many validators abstain or misbehave (Leonardos et al., 2019).

In intraday trading, the same high-level idea is adapted to delayed labels and domain-specific utility. Numin combines eight base models by weighted majority voting over five-minute candlestick data, but updates weights by an exponential moving average of windowed performance scores rather than by per-example multiplicative penalties: $w_{t,i}\ge 0$7 The weighting statistic $w_{t,i}\ge 0$8 is either accuracy or a utility score derived from a fixed utility matrix. Because labels arrive with a ten-round delay, the update window ends at $w_{t,i}\ge 0$9 to avoid look-ahead bias. On a five-day average, the utility-weighted ensemble with window $0/1$0 achieved $0/1$1 accuracy and utility $0/1$2, outperforming the individual models in that setting; the paper explicitly notes, however, that this windowed EMA adaptation does not carry standard mistake or regret guarantees (Mukherjee et al., 2024).

Across these literatures, the main limitations are setting-specific rather than universal. Distributed WMA analyses assume synchronous rounds, connected communication graphs, and typically a complete graph in theory; the game-theoretic and trust-estimation analyses rely on conditional independence; PAC-Bayesian majority-vote bounds assume i.i.d. data and a data-independent prior; and several empirical adaptations replace classical multiplicative penalties by task-driven heuristics, thereby forfeiting the canonical WMA mistake or regret guarantees (Ouyang et al., 2011, Georgiou et al., 2013, Masegosa et al., 2020).