When Can Voting Help, Hurt, or Change Course? Exact Structure of Binary Test-Time Aggregation

Abstract: Majority voting is one of the few black-box interventions that can improve a fixed stochastic predictor: repeated access can be cheaper than changing a high-capability model. Classical fixed-competence theory makes this intervention look monotone -- more votes help above the majority threshold and hurt below it. We show that this picture is fundamentally incomplete. Under the de Finetti representation for exchangeable repeated correctness, voting is governed by a latent distribution of per-example correctness probabilities. Even simple latent mixtures can generate sharply different voting curves, including nonmonotone behavior and, in an explicit construction, infinitely many trend changes. The full latent law determines the curve, but the curve does not determine the law. The exact object recovered by voting is a signed voting signature: at each binomial variance scale, it records excess latent mass above rather than below the majority threshold. Our main theorem proves that the complete odd-budget curve and this signature are equivalent: the curve increments are signed Hausdorff moments, and the full curve recovers the signature uniquely. This viewpoint explains shape phenomena, branch-symmetric nonidentifiability, realizability, variation, and endpoint rates. It also separates estimation regimes: direct per-example success-probability information targets the full signature, whereas fixed-depth grouped labels reveal only a finite prefix.

• The paper demonstrates that binary majority voting can exhibit nonmonotone and oscillatory behavior, countering traditional fixed-competence theories.
• It introduces the signed voting signature to precisely characterize how latent heterogeneity influences vote aggregation and convergence properties.
• Empirical findings reveal that the effectiveness of aggregation depends critically on instance difficulty and the distribution of per-vote correctness.

Exact Structure and Behavior of Binary Test-Time Voting Aggregation

Introduction

Majority voting is a commonly employed technique for improving the reliability of stochastic prediction systems at inference time—such as LLM sampling, ensemble methods, and various randomized black-box models. Traditionally, classic fixed-competence theory suggests that increasing the number of votes monotonically enhances accuracy above the majority threshold and degrades it below. However, this paper, "When Can Voting Help, Hurt, or Change Course? Exact Structure of Binary Test-Time Aggregation" (2605.05592), challenges that perspective by analyzing majority voting under exchangeability and latent heterogeneity. The work reveals that binary voting aggregation can display markedly nonmonotone and even oscillatory behavior, fully characterizes the structural object recovered by the voting curve, and rigorously dissects the implications for estimation and practical deployment.

Binary Voting under Latent Heterogeneity

The classical analysis of majority voting assumes identical, independent correctness probabilities across votes. Under this homogeneity, the Condorcet Jury Theorem applies: if the per-vote correctness $q > 1/2$, majority voting with an increasing number of votes drives error toward zero; if $q < 1/2$, it amplifies errors. However, in realistic scenarios—especially in machine learning applications with black-box models or datasets containing a variety of instance difficulties—this assumption fails.

By invoking the de Finetti theorem for infinite exchangeable sequences, this work introduces a latent variable $Q$, representing the per-example correctness probability drawn from an unknown distribution $\Pi$. Each example's repeated calls are i.i.d. conditioned on $Q$ but $Q$ can vary across the population. As a result, the accuracy of odd-budget majority voting becomes:

$$V_n = E[P_n(Q)], \quad P_n(q) = P\{Bin(2n+1, q) \geq n+1\}$$

where $V_0 = E[Q]$ is the single-sample accuracy.

Empirical Voting Curve Phenomena

(Across textual datasets and machine learning settings, instances often contain a mixture of easy, ambiguous, and misleading examples.) The authors establish that simple discrete mixtures for the latent law $\Pi$ can generate a wide diversity of voting curve behaviors for $V_n$ as a function of $q < 1/2$0:

• Majority voting can yield flat, rapidly decreasing, dip-then-ascend, peak-then-fall, or slow-rise curves, all with identical one-vote accuracy $q < 1/2$1.
• There can exist latent distributions where the majority-vote accuracy curve changes direction infinitely often as the vote budget increases.

This complexity is illustrated vividly in Figure 1, where each curve corresponds to a different latent law with identical $q < 1/2$2.

Figure 1: Different behaviors of voting curves as a function of vote budget, emanating from simple discrete latent mixtures, all with $q < 1/2$3.

Such behavior contradicts the monotonicity intuition emanating from traditional fixed-competence scenarios and demonstrates that hidden instance heterogeneity fundamentally alters how test-time aggregation should be interpreted.

Structural Characterization: The Signed Voting Signature

A key contribution is the explicit characterization of what aspects of the latent law $q < 1/2$4 can be identified from the full sequence $q < 1/2$5. The authors define the signed voting signature $q < 1/2$6, a signed measure on $q < 1/2$7:

$q < 1/2$8

This signature is:

• Supported on the binomial variance scale $q < 1/2$9.
• Weighted by the orientation $Q$0: above-threshold ($Q$1) mass helps, below-threshold mass hurts.
• Invariant under branch-symmetric transformations ($Q$2 with no net effect on vote aggregation).

Main Theorem: The sequence of vote increments $Q$3 are the signed Hausdorff moments of $Q$4, and $Q$5 is uniquely determined by (and uniquely determines) the full odd-budget voting curve. However, the full latent law $Q$6 is not identified; only this quotient is.

This aligns the estimation and extrapolation of vote curves with the classical (signed) moment problem and exposes the precise source of non-identifiability.

Calculus of Voting Curves: Shape, Rates, and Endpoint Analysis

Leveraging the signed voting signature enables fine-grained analysis of curve shape and convergence properties:

• The sign and support of $Q$7 determine whether $Q$8 is monotonic, convex, or oscillatory.
• If $Q$9 is nonnegative, $\Pi$0 is nondecreasing and discretely concave; if nonpositive, monotone and convex; mixed sign allows arbitrary behavior, including infinitely many trend reversals.
• The endpoint accuracy $\Pi$1 (as the vote budget tends to infinity) is:

$\Pi$2

• Budget-to-budget variation is rapidly suppressed if $\Pi$3 is supported near $\Pi$4 (i.e., the population concentrates on very easy or very hard cases), but persists if there is mass near $\Pi$5 (hard-to-decide cases).
• The rate at which $\Pi$6 is sharp and controlled by how much latent mass is near $\Pi$7, with polynomial-exponent lower and upper bounds.

Estimating the Signed Signature in Practice

The identifiability and estimation of $\Pi$8 depend crucially on the nature of validation access:

• Per-example correctness probabilities $\Pi$9 available: $Q$0 can be estimated as an empirical signed pushforward, ensuring nonparametric consistency.
• Grouped repeated labels per example (repeat depth $Q$1): only the first $Q$2 signature moments are identified, creating a finite-dimensional nonparametric estimation problem. Fixed-depth nonparametric MLE identifies only this prefix.
• Extrapolation beyond observed repeat depth is statistically ill-posed without additional structure or growing $Q$3.

Thus, stable estimation of full long-range voting behavior in the black-box setting demands structural prior knowledge (e.g., smoothness, sparsity, or parametric assumptions) or growing validation budgets.

Implications and Future Directions

This structural analysis of binary test-time voting has several implications:

• Practical Performance Extraction: Blindly increasing the number of majority votes may not monotonically improve accuracy and, without knowledge of the latent signature, might even degrade performance under certain latent mixtures.
• Evaluation and Benchmarking: Comparing systems purely on one-vote accuracy can be misleading; analyses should consider the full voting curve or its identifiable prefix as a function of repeat trials.
• Estimation Protocols: For robust estimation of majority-vote improvement, validation datasets must either provide deep repeat access or regularized structural modeling.
• Generalization Limits: The results show that non-binary (multiclass, plurality) settings do not generally admit one-dimensional signed signature invariants—higher-dimensional latent geometry is required.

Unanswered questions include establishing structural constraints (e.g., unimodality or log-concavity of the latent density) yielding more regular behavior, and extending the signature calculus to multiclass scenarios beyond symmetry assumptions.

Conclusion

This paper rigorously decomposes the aggregation dynamics of majority voting in stochastic prediction settings, precisely characterizing when voting helps, hurts, or reverses. The signed voting signature $Q$4 encapsulates all the aggregate information in the population that majority voting can recover. Identifiability limitations and estimation boundaries are made explicit, grounding practical test-time aggregation in a concrete mathematical framework. These insights guide both theoretical and empirical practice in the evaluation and extraction of model performance via repeated querying and aggregation.