---
title: Pass@$k$ Paradox in LLM Evaluation
url: https://www.emergentmind.com/topics/pass-k-paradox
type: topic
---

# Pass@$k$ Paradox in LLM Evaluation

The Pass@$k$ paradox concerns the widespread use of the pass@$k$ metric in evaluating and optimizing large language models (LLMs) for tasks with verifiable rewards, such as mathematical reasoning and code synthesis. While pass@$k$—the probability that at least one of $k$ independently sampled outputs is correct—was adopted due to its intuitive appeal for multi-sample evaluation, its direct optimization and even its use for ranking have been shown to have subtle failures. These include unstable or misleading model comparisons, vanishing learning signals during training, exploration collapse, and statistical paradoxes in practice. The paradox is now recognized as both a challenge for RL with verifiable rewards and an opportunity for theoretical clarification and improved algorithm design.

## 1. Definition of Pass@$k$ and Metric Formulation

Given an autoregressive LLM policy $\pi_\theta(y|x)=\prod_{t=1}^T\pi_\theta(a_t|x,a_{<t})$ and a deterministic verifier $V(x,y)\in\{0,1\}$, the pass@$k$ probability is
\[
J_k(x; \theta) = 1 - \left(1 - J_1(x; \theta)\right)^k,
\]
where $J_1(x;\theta)=\mathbb{E}_{y\sim\pi_\theta}[V(x, y)]$ is the expected single-sample accuracy. Aggregating across a dataset $\mathcal{D}$ yields $J_k(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[J_k(x; \theta)\right]$ [2511.16231]. Pass@$k$ thus reflects the success probability in $k$ draws, and is a nonlinear, non-differentiable function of model outputs.

In practical settings with $n \gg k$ samples per problem and $c$ observed correct samples, the unbiased estimator is
\[
\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}
\]
[2510.04265, 2510.14807].

## 2. Theoretical Analysis: Gradient Structure and Learning Signal

The pass@$k$ objective is not an independent optimization direction. Applying the chain rule,
\[
\nabla_\theta J_k(x; \theta) = k (1 - J_1(x; \theta))^{k-1} \nabla_\theta J_1(x; \theta) \equiv \alpha_k(x, \theta) \nabla_\theta J_1(x; \theta)
\]
with scaling factor $\alpha_k(x, \theta) = k (1 - J_1(x; \theta))^{k-1} \in [0, k]$ [2511.16231]. This means optimizing pass@$k$ is a collinear reweighting of the pass@$1$ gradient. Crucially, it does not introduce new search directions or incentivize the discovery of alternative correct modes.

Two degenerate learning regimes result:
- **Failure regime** ($J_1 \to 0$): Large $\alpha_k$ in theory, yet empirical gradients vanish as correct samples are nearly impossible to discover, so optimization stalls exactly where exploration is most needed.
- **Saturation regime** ($J_1 \to 1$): $\alpha_k \to 0$, so the objective becomes uninformative, precluding further improvements in policy robustness or diversity [2511.16231, 2510.23049].

## 3. Exploration Collapse and Mode Concentration

Policy gradient RLVR iteratively magnifies discovered modes at the expense of the others. Given a correct solution partitioned into modes $M_1$ (found) and $M_2$ (unfound), as mass $\pi_t(M_2) < \epsilon \ll 1$, the probability $P(\text{discover } M_2) \approx k\epsilon$ becomes negligible. Consequently, policy updates reinforce $M_1$ only:
- The pass@$1$ probability, $p_t = \pi_t(M_1)$, grows toward unity.
- pass@$k$ saturates at $1 - (1-p_t)^k$.
- As $p_t \to 1$, the gap $\Delta_t(k) = \text{pass@}k - p_t$ vanishes, so pass@$k$ converges to pass@$1$.

This "exploration collapse" reflects entropy shrinkage and diversity loss. Empirically, RLVR with vanilla PPO-style optimization shows this effect as increased pass@$1$ but degraded or stagnant pass@$k$ for $k > 1$ [2510.14807].

## 4. Empirical Manifestations: The Pass@$k$ Paradox in Practice

The pass@$k$ paradox manifests both in model optimization and in evaluation/ranking:
- **Optimization**: RLVR methods (e.g., GRPO, PPO) improve pass@$1$ while reducing pass@$k$ as training proceeds—sharpening the output distribution to maximize the top-1 likelihood at the cost of alternative plausible answers (mode collapse). SimKO proposes an asymmetric update rule (boosting top-$K$ on correct, penalizing top-1 on incorrect at high entropy steps) to mitigate this, empirically raising pass@$k$ across LLMs and benchmarks [2510.14807].
- **Evaluation and ranking**: pass@$k$ is highly sensitive to sample count and correct-answer count, especially for moderate $n,k$. Its nonlinear dependence on $c$ creates large sampling variance, which can invert model rankings, producing unreliable or misleading leaderboards [2510.04265]. Empirical studies demonstrate that as $k$ increases (or $n$ decreases), fluctuations and order-inversions become more likely even if one model is truly superior in underlying accuracy.

A Bayesian evaluation framework using posterior means and credible intervals over success rates gives substantially more stable, statistically interpretable rankings, and avoids these pitfalls [2510.04265].

## 5. Mathematical and Probabilistic Connections

The combinatorial structure of pass@$k$ mirrors classical problems in probability. For instance, the "lost boarding pass" paradox generalizes as follows: The probability that the $m$th passenger (where $m > k$) sits in their assigned seat is
\[
P(\text{seat } m \text{ occupied}) = \frac{k}{n-m+k+1}
\]
and thus
\[
P(\text{passenger } m \text{ in own seat}) = \frac{n-m+1}{n-m+k+1}
\]
[1910.02515]. Independence of occupancy events induces a symmetry that underpins the form of pass@$k$ probabilities and their stochastic properties. In the large $n$ limit, the empirical cycle structure converges to the Poisson–Dirichlet law, providing a connection between pass@$k$ anomalies and deep results in random permutation theory.

## 6. Algorithmic and Statistical Remedies

Analysis of pass@$k$ paradox consequences guides mitigation strategies:
- **Exploration encouragement**: Explicit incentives (e.g., entropy bonuses, submodular diversity, set-based objectives) must be introduced during RLVR, as optimizing pass@$k$ itself is insufficient. Algorithms such as SimKO and reward-level regularization via surrogate objectives can partially restore exploration and sustain pass@$k$ without sacrificing pass@$1$ [2510.14807, 2510.23049].
- **Evaluation best practices**: Bayesian inference over success probabilities, with Dirichlet or Beta priors, yields closed-form posterior means and credible intervals (e.g., $\hat{\theta} = (1 + n_\text{succ})/(2+N)$ for the uniform Beta(1,1) prior), enabling robust, rapid-converging model comparisons and principled uncertainty quantification [2510.04265].

## 7. Implications and Outlook

The pass@$k$ paradox highlights a fundamental gap between metrics suitable for model evaluation versus those fit for optimization:
- pass@$k$ is effective as a diagnostic tool for latent diversity and multi-sample performance but fails as an RL objective due to vanishing or misguided learning signals.
- Both theoretical and empirical evidence recommend decoupling pass@$k$ measurement from model training; instead, adopt explicit exploration mechanisms and rigorous, uncertainty-calibrated evaluation frameworks.
- Surrogate reward maximization and posterior-based evaluation now constitute standard methodologies in response to the paradox [2511.16231, 2510.23049, 2510.04265].

Ongoing research continues to refine both the exploration incentives necessary in RLVR and the statistical tools for reliable, compute-efficient benchmarking under limited sampling regimes. The resolution of the pass@$k$ paradox thus informs the design of robust reasoning agents and the criteria by which their progress is measured.

Source: https://www.emergentmind.com/topics/pass-k-paradox