Pass@$k$ Paradox in LLM Evaluation

• Pass@$k$ paradox is the phenomenon where using the metric to measure the probability of at least one correct sample leads to unstable learning signals and statistical anomalies.
• The paradox manifests in both optimization and evaluation, causing improved top-1 accuracy to mask reduced diversity and misleading model comparisons.
• Statistical and algorithmic remedies, such as Bayesian evaluations and explicit exploration bonuses, are proposed to mitigate these issues and enhance LLM robustness.

The Pass@$k$ paradox concerns the widespread use of the pass@$k$ metric in evaluating and optimizing LLMs for tasks with verifiable rewards, such as mathematical reasoning and code synthesis. While pass@$k$—the probability that at least one of $k$ independently sampled outputs is correct—was adopted due to its intuitive appeal for multi-sample evaluation, its direct optimization and even its use for ranking have been shown to have subtle failures. These include unstable or misleading model comparisons, vanishing learning signals during training, exploration collapse, and statistical paradoxes in practice. The paradox is now recognized as both a challenge for RL with verifiable rewards and an opportunity for theoretical clarification and improved algorithm design.

1. Definition of Pass@$k$ and Metric Formulation

Given an autoregressive LLM policy $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$ and a deterministic verifier $V(x,y)\in\{0,1\}$, the pass@$k$ probability is

$$J_k(x; \theta) = 1 - \left(1 - J_1(x; \theta)\right)^k,$$

where $$J_1(x;\theta)=\mathbb{E}_{y\sim\pi_\theta}[V(x, y)]$$ is the expected single-sample accuracy. Aggregating across a dataset $k$0 yields $k$1 (Yu, 20 Nov 2025). Pass@$k$2 thus reflects the success probability in $k$3 draws, and is a nonlinear, non-differentiable function of model outputs.

In practical settings with $k$4 samples per problem and $k$5 observed correct samples, the unbiased estimator is

$k$6

(Hariri et al., 5 Oct 2025, Peng et al., 16 Oct 2025).

2. Theoretical Analysis: Gradient Structure and Learning Signal

The pass@$k$7 objective is not an independent optimization direction. Applying the chain rule,

$k$8

with scaling factor $k$9 (Yu, 20 Nov 2025). This means optimizing pass@$k$0 is a collinear reweighting of the pass@$k$1 gradient. Crucially, it does not introduce new search directions or incentivize the discovery of alternative correct modes.

Two degenerate learning regimes result:

• Failure regime ($k$2): Large $k$3 in theory, yet empirical gradients vanish as correct samples are nearly impossible to discover, so optimization stalls exactly where exploration is most needed.
• Saturation regime ($k$4): $k$5, so the objective becomes uninformative, precluding further improvements in policy robustness or diversity (Yu, 20 Nov 2025, Thrampoulidis et al., 27 Oct 2025).

3. Exploration Collapse and Mode Concentration

Policy gradient RLVR iteratively magnifies discovered modes at the expense of the others. Given a correct solution partitioned into modes $k$6 (found) and $k$7 (unfound), as mass $k$8, the probability $k$9 becomes negligible. Consequently, policy updates reinforce $k$0 only:

• The pass@$k$1 probability, $k$2, grows toward unity.
• pass@$k$3 saturates at $k$4.
• As $k$5, the gap $k$6 vanishes, so pass@$k$7 converges to pass@$k$8.

This "exploration collapse" reflects entropy shrinkage and diversity loss. Empirically, RLVR with vanilla PPO-style optimization shows this effect as increased pass@$k$9 but degraded or stagnant pass@$k$0 for $k$1 (Peng et al., 16 Oct 2025).

4. Empirical Manifestations: The Pass@$k$2 Paradox in Practice

The pass@$k$3 paradox manifests both in model optimization and in evaluation/ranking:

• Optimization: RLVR methods (e.g., GRPO, PPO) improve pass@$k$4 while reducing pass@$k$5 as training proceeds—sharpening the output distribution to maximize the top-1 likelihood at the cost of alternative plausible answers (mode collapse). SimKO proposes an asymmetric update rule (boosting top-$k$6 on correct, penalizing top-1 on incorrect at high entropy steps) to mitigate this, empirically raising pass@$k$7 across LLMs and benchmarks (Peng et al., 16 Oct 2025).
• Evaluation and ranking: pass@$k$8 is highly sensitive to sample count and correct-answer count, especially for moderate $k$9. Its nonlinear dependence on $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$0 creates large sampling variance, which can invert model rankings, producing unreliable or misleading leaderboards (Hariri et al., 5 Oct 2025). Empirical studies demonstrate that as $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$1 increases (or $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$2 decreases), fluctuations and order-inversions become more likely even if one model is truly superior in underlying accuracy.

A Bayesian evaluation framework using posterior means and credible intervals over success rates gives substantially more stable, statistically interpretable rankings, and avoids these pitfalls (Hariri et al., 5 Oct 2025).

5. Mathematical and Probabilistic Connections

The combinatorial structure of pass@$$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$3 mirrors classical problems in probability. For instance, the "lost boarding pass" paradox generalizes as follows: The probability that the $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$4th passenger (where $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$5) sits in their assigned seat is

$$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$6

and thus

$$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$7

(Grimmett et al., 2019). Independence of occupancy events induces a symmetry that underpins the form of pass@$$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$8 probabilities and their stochastic properties. In the large $$\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$$9 limit, the empirical cycle structure converges to the Poisson–Dirichlet law, providing a connection between pass@$V(x,y)\in\{0,1\}$0 anomalies and deep results in random permutation theory.

6. Algorithmic and Statistical Remedies

Analysis of pass@$V(x,y)\in\{0,1\}$1 paradox consequences guides mitigation strategies:

• Exploration encouragement: Explicit incentives (e.g., entropy bonuses, submodular diversity, set-based objectives) must be introduced during RLVR, as optimizing pass@$V(x,y)\in\{0,1\}$2 itself is insufficient. Algorithms such as SimKO and reward-level regularization via surrogate objectives can partially restore exploration and sustain pass@$V(x,y)\in\{0,1\}$3 without sacrificing pass@$V(x,y)\in\{0,1\}$4 (Peng et al., 16 Oct 2025, Thrampoulidis et al., 27 Oct 2025).
• Evaluation best practices: Bayesian inference over success probabilities, with Dirichlet or Beta priors, yields closed-form posterior means and credible intervals (e.g., $V(x,y)\in\{0,1\}$5 for the uniform Beta(1,1) prior), enabling robust, rapid-converging model comparisons and principled uncertainty quantification (Hariri et al., 5 Oct 2025).

7. Implications and Outlook

The pass@$V(x,y)\in\{0,1\}$6 paradox highlights a fundamental gap between metrics suitable for model evaluation versus those fit for optimization:

• pass@$V(x,y)\in\{0,1\}$7 is effective as a diagnostic tool for latent diversity and multi-sample performance but fails as an RL objective due to vanishing or misguided learning signals.
• Both theoretical and empirical evidence recommend decoupling pass@$V(x,y)\in\{0,1\}$8 measurement from model training; instead, adopt explicit exploration mechanisms and rigorous, uncertainty-calibrated evaluation frameworks.
• Surrogate reward maximization and posterior-based evaluation now constitute standard methodologies in response to the paradox (Yu, 20 Nov 2025, Thrampoulidis et al., 27 Oct 2025, Hariri et al., 5 Oct 2025).

Ongoing research continues to refine both the exploration incentives necessary in RLVR and the statistical tools for reliable, compute-efficient benchmarking under limited sampling regimes. The resolution of the pass@$V(x,y)\in\{0,1\}$9 paradox thus informs the design of robust reasoning agents and the criteria by which their progress is measured.