When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

Abstract: People overthink; LLMs over-sample, and the extra effort can talk both into a worse answer. Reasoning systems answer a hard question by sampling it many times (test-time scaling), and the more they draw, the more often a correct answer turns up somewhere, so coverage, the fraction of problems with at least one correct try, climbs and appears to be progress. But a deployed system must return one answer, and choosing it, not knowing which try is right, is selection; selection is capped, and past a point extra samples only make the model surer of a confident mistake, even as every draw adds cost. The gap between climbing coverage and stalled selection, the identifiability gap, is the answer a model can produce but not pick. So the real question is not whether to sample but how far, and the answer is: not far. For picking an answer, the vote has already settled within a few dozen draws, the modal ceiling; for scoring a benchmark, sooner still, the correlation ceiling. Beyond that, extra draws cost compute and add nothing, and can even make the answer worse. This paper turns the cutoff into a single number, the effective number of samples, that any sampling run already reveals. The bottleneck is recognizing a right answer, not generating one.

• The paper formalizes that repeated test-time sampling in LLMs leads to a 'modal ceiling' where selection performance plateaus, highlighting diminishing returns.
• It introduces the effective sample size (nₑff) which accounts for intrinsic output correlation (ρ), showing that additional samples often provide redundant information.
• Empirical findings reveal a significant identifiability gap between coverage and selection, recommending improved decorrelation techniques and better selection strategies.

The Modal Ceiling and Correlation Ceiling in Test-Time Sampling for LLMs

Theoretical Foundations: Test-Time Scaling as Cluster Sampling

The paper "When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling" (2606.28661) formalizes the limitations inherent in test-time scaling for LLMs, focusing on repeated sampling as a lever for improved output quality. It rigorously distinguishes between coverage (probability any sample is correct) and selection (probability the selected/plurality answer is correct), demonstrating that increased sampling delivers rapidly diminishing returns for selection, and, critically, can actively degrade performance past a certain point due to the modal ceiling phenomenon.

The key insight is that test-time sampling forms "clusters": each repeated attempt for a fixed prompt constitutes a cluster of correlated draws, not independent samples. The intraclass correlation $\rho$ among correct/incorrect indicators within one problem is central, manifesting as a design effect that limits statistical power. The effective number of independent samples is $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$—a well-known correction from the survey-sampling literature, here newly applied to LLM inference.

Figure 1: The dual structure of test-time sampling—(a) the divergence between coverage and selection, and (b) how the effective number of samples $n_{\mathrm{eff}}$ saturates due to intraclass correlation.

Coverage, Selection, and the Identifiability Gap

Coverage (usually pass@$n$) increases monotonically with the number of samples, only saturating if the model has zero probability of generating a correct answer. This is true as long as the correct answer is within the support of the conditional answer distribution for a given prompt. However, coverage does not address which answer to actually return when no perfect verifier exists at deployment.

Selection (via majority/plurality/self-consistency or best-of-$n$ with reward models) converges to a "modal ceiling"—the probability that the most frequently generated answer for a problem is correct. More sampling, past a certain point, only entrenches the selection of an answer that could be wrong if it dominates the distribution, a phenomenon termed anti-scaling. Figure 2 exemplifies a scenario where coverage detects a correct answer among samples, but selection fails since the majority answer is incorrect.

Figure 2: On challenging problems, coverage identifies the presence of a correct answer among samples, but the plurality vote can confidently return an incorrect answer if the correct one is not most frequent.

The identifiability gap is defined as the difference between coverage and selection: it represents the set of problems for which a correct answer is generated but cannot be reliably selected under typical selection strategies.

The Correlation Ceiling and Marginal Value of Sampling

For benchmark mean estimation, repeated sampling is bottlenecked by the correlation ceiling. The effective number of samples $n_{\mathrm{eff}}$ plateaus at $1/\rho$, making large sample budgets highly redundant when $\rho$ is non-negligible.

Figure 3: Effective sample size for several $\rho$ values; for $\rho = 0.1$, $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$0 saturates near 10, regardless of the nominal sample count.

The marginal value of additional samples sharply decays as $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$1 beyond the ceiling, so further costs are not justified for estimation purposes.

Figure 4: Marginal gain for each additional sample decays quadratically, quickly approaching zero past $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$2.

Empirically, reported intraclass correlations for popular benchmarks and models (e.g., Llama-3-8B/70B on GSM8K and MATH) lie in the $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$3–$n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$4 range, meaning $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$5 samples per problem are statistically worth only about $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$6 effective draws.

Power Law in Coverage, Mode Collapse in Selection

When per-problem difficulty is heavy-tailed and success rates $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$7 are drawn from Beta distributions, coverage loss $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$8 decays as a power law, in contrast to the exponential decay predicted under i.i.d. assumptions. This yields much slower improvements—and with mode collapse (atomized $n_{\mathrm{eff}} = n/(1 + (n-1)\rho)$9 at $n_{\mathrm{eff}}$0), no number of samples ever reaches all problems.

Figure 5: Both exponential and power-law coverage regimes are shown; heavy-tailed difficulty yields slow, log-linear improvement in coverage.

In contrast, selection (self-consistency, majority/plurality voting) is capped by the probability that the mode of the answer distribution matches the ground truth. Crucially, as $n_{\mathrm{eff}}$1 selection does not improve beyond this mode hit rate, and may anti-scale if the modal answer is incorrect but reachable.

Figure 6: Left: when the most common answer is correct, coverage and selection coincide at 1. Right: when it is wrong, selection falls while coverage rises, illustrating anti-scaling.

The binary special case (majority voting between correct/incorrect candidates) reduces the modal ceiling to the probability that the per-attempt correctness exceeds $n_{\mathrm{eff}}$2, but empirical selection rates are bounded above this by the scenario where correct answers dominate even when sub-majority.

Figure 7: Binary majority-vote accuracy plateaus below 1 for correlated samples; real data often achieves a higher selection ceiling due to error dispersion.

Empirical Evidence: Identifiability Gap in Practice

Analysis of large sampling logs (e.g., [brown2024monkeys]) shows a consistent and substantial coverage-selection gap. For instance, with Llama-3-8B on GSM8K, coverage reaches $n_{\mathrm{eff}}$3 while self-consistency plateaus at $n_{\mathrm{eff}}$4, indicating a $n_{\mathrm{eff}}$5 identifiability gap—these are solvable problems where the correct answer cannot be surfaced via selection.

Figure 8: Empirical coverage and self-consistency curves illustrate the persistent gap in Llama-3-8B on GSM8K; the shaded region is the identifiability gap.

On dependent-draw logs (e.g., [beeching2024scaling]), the modal ceiling can be even more severe: coverage approaches $n_{\mathrm{eff}}$6, but self-consistency saturates at $n_{\mathrm{eff}}$7. Again, sample plurality converges to the wrong answer on more than half the problems.

Figure 9: Selection gap under dependent draws on MATH-500 with Llama-3.2-1B. The effective number of unique answers is approximately 13, but selection cannot surpass $n_{\mathrm{eff}}$8 accuracy.

Practical Implications and Recommendations

The implications for both evaluation and deployment are significant:

• For evaluation: Estimation of mean accuracy over a benchmark should use $n_{\mathrm{eff}}$9 (not the nominal $n$0). Adding more samples per problem past the ceiling is wasteful—budget should be used to increase the number of problems instead.
• For selection (deployed inference): There is little to no benefit from increasing $n$1 once the selection curve saturates (typically after a few dozen samples). Additional samples can increase confidence in the wrong answer (anti-scaling).
• For coverage (with a perfect/verifiable solution): Sampling remains beneficial, as each additional draw increases the chance of discovering a correct answer; there is no within-problem ceiling.
• For model and protocol design: De-biasing the answer distribution, decorrelating samples (via higher temperature, prompt diversity, etc.), or improving selection mechanisms have greater leverage than increasing the raw sample count.

The paper advocates reporting the effective sample size and mode hit rate in any empirical evaluation and urges precise measurement of $n$2 in practice to ground claims of statistical significance.

Conclusion

This work uncovers that the main impediment in modern test-time scaling with LLMs is not in generating correct answers but in reliably surfacing them for deployment when lacking a reliable verifier. The analysis of the modal and correlation ceilings demonstrates that repeated sampling rapidly becomes redundant for both statistical estimation and modal selection as a result of nontrivial intraclass correlation and distribution collapse. The gap between coverage and selection—the identifiability gap—is robust, empirically substantial, and cannot be remedied by more naive sampling alone. Future advancements should focus on deconcentrating the answer distribution, constructing better selection methods, and reconsidering the primary axes of compute allocation in large-scale LLM inference.