---
title: 'Pass@k: Evaluation & Insights'
url: https://www.emergentmind.com/topics/pass-k-metric-f1419870-1396-4930-8901-987e892e30e3
type: topic
---

# Pass@k: Evaluation & Insights

The Pass@k metric, also known as Pass-at-k, quantifies the probability that at least one out of k independent samples from a model yields a correct solution to a given problem. It has become a standard evaluation method for large language models (LLMs), especially in coding, mathematical reasoning, and other discrete output tasks. Pass@k serves as a bridge between pure accuracy (pass@1) and broader exploration, but is susceptible to misinterpretations when applied outside its intended sampling regime. Recent research has both extended its mathematical foundations and highlighted its limitations, motivating alternative metrics such as Cover@τ.

## 1. Formal Definition and Mathematical Properties

Given a test suite of $T$ problems, let $p_i \in [0,1]$ denote the probability that a single sample from a model will solve problem $i$. The Pass@k for model $M$ and budget $k$ is:
\[
\mathrm{Pass@}k = \frac{1}{T} \sum_{i=1}^T \Pr[\,\text{≥1 success in }k\text{ trials on }i] = \frac{1}{T} \sum_{i=1}^T \Bigl[1 - (1 - p_i)^k\Bigr]
\]
For a single problem, the metric reduces to $1 - (1-p)^k$, where $p$ is the per-sample success probability. When multiple completions ($k$) are drawn independently, Pass@k reflects the chance that at least one sample yields success.

As $k \rightarrow \infty$, $\mathrm{Pass@}k \rightarrow 1$ for any $p_i > 0$, indicating that in the large-$k$ limit, the metric saturates regardless of the true difficulty or reliability of the underlying model on each problem.

## 2. Statistical Estimation and Practical Computation

In practice, Pass@k can be computed either analytically (with access to exact $p_i$) or as an unbiased estimator when $n \ge k$ samples have been drawn:
\[
\widehat{\mathrm{Pass@}k} = \frac{1}{T}\sum_{i=1}^T \left[1 - \frac{\binom{n-c_i}{k}}{\binom{n}{k}}\right]
\]
with $c_i$ the number of correct samples among $n$ runs for problem $i$. This estimator is used widely in code generation and reasoning benchmarks.

Notably, the high variance of $\widehat{\mathrm{Pass@}k}$ in regimes where $n \approx k$ or when $T$ is small can yield unstable or misleading results, as emphasized in [2510.04265]. It is common to use a large $n$ (e.g., 256 or 300) to obtain stable estimates, but this is often computationally expensive.

## 3. Interpretation: Breadth vs. Depth and the Crossover Phenomenon

Pass@k is often interpreted as a "breadth" metric: it rewards any problem that can be solved at least once across $k$ samples, even if the success is due to random chance rather than robust reasoning. At low $k$ (e.g., $k=1$), Pass@k measures average per-sample success, tightly coupling to model depth and reliability. At large $k$, it increasingly reflects whether a model's probability mass on the correct answer is nonzero, regardless of its magnitude.

A key empirical observation ("crossover phenomenon" [2510.08325]): RL-fine-tuned models typically outperform base models on Pass@k at small $k$, but are overtaken by the base model as $k$ increases and random guessing dominates. On problems with discrete answer spaces, a base model can eventually enumerate the correct response at large $k$, yielding high Pass@k that overstates its true reasoning ability.

## 4. Cover@τ: Reliability-Thresholded Generalization

To address the reliability ambiguity in Pass@k, [2510.08325] proposes Cover@τ:
\[
\mathrm{Cover@}\tau = G(\tau) = \frac{1}{T}\sum_{i=1}^T \mathbf{1}\{p_i \geq \tau\}
\]
This measures the fraction of problems where the model's per-sample success probability is at least $\tau$, explicitly parameterizing the reliability requirement. $G(0^+)$ corresponds to the problems with $p_i > 0$ (ever solved), while $G(1)$ captures the problems solved almost always when sampled. 

Theoretical connections relate the two:
\[
\mathrm{Pass@}k = \int_{0}^{1} k(1-\tau)^{k-1} G(\tau)\, d\tau
\]
Thus, Pass@k can be interpreted as a (Beta-distributed) weighted average over Cover@τ, with the weight concentrating near $\tau=0$ as $k$ increases (i.e., emphasizing breadth over depth).

## 5. Applications in Policy Optimization and RLVR

Pass@k is both an evaluation metric and an objective for direct optimization in reinforcement learning with verifiable rewards (RLVR). Recent work such as Pass@K Policy Optimization (PKPO) [2505.15201] and Pass@k Training [2508.10751] derive unbiased, low-variance estimators for Pass@k and its policy gradient, enabling principled sample-level reward transformations. Core steps include:
- For each problem: draw $n \ge k$ completions, score their correctness.
- Compute Pass@k unbiasedly across all $k$-sized subsets (using combinatorial estimators).
- Transform per-sample rewards to optimize for the best of $k$ over all subsets (joint rather than independent utility).
- Anneal $k$ (e.g., $k$ starts large to prioritize exploration, then decreases to focus on exploitation), as this empirically lifts both Pass@1 and Pass@k on challenging tasks.

Empirical results demonstrate that reward transformations and advantage shaping for Pass@k—as in [2510.23049], which unifies direct REINFORCE and GRPO-style approaches—yield robust improvements, particularly in hard or low-entropy settings. Moreover, algorithms such as SimKO [2510.14807] explicitly counteract the probability-concentration effect, improving the diversity of reasoning paths and boosting Pass@k relative to vanilla RLVR methods.

## 6. Limitations, Misuses, and Bayesian Alternatives

Pass@k, while intuitive, is prone to several statistical pitfalls in standard practice:
- **Variance and ranking instability**: For small $n$ or $T$, Pass@k has high variance, producing inconsistent rankings as $k$ varies or between runs [2510.04265].
- **No confidence intervals**: Pass@k does not yield analytic uncertainty estimates, requiring computationally intensive bootstrap for CIs.
- **Ranking paradoxes**: Model A may win for $k=2$ but lose at $k=4$, or vice versa, when $n$ is small.

A Bayesian alternative, Bayes@N, models evaluation outcomes as categorical with Dirichlet priors, enabling closed-form posterior means and credible intervals [2510.04265]. Under a uniform prior, Bayes@N yields rankings equivalent to average accuracy (Pass@1), but with principled uncertainty and robustness to small-sample effects. The approach generalizes seamlessly to graded/partial-credit rubrics, prior integration, and is recommended for stable LLM evaluation.

| Metric       | Convergence | CIs | Prior | Categorical |
|--------------|-------------|-----|-------|-------------|
| Pass@k       | slow, unstable | no  | no    | no          |
| avg@N        | medium        | boot/approx | no | no      |
| Bayes@N      | fast, stable | yes | yes   | yes         |

## 7. Extensions, Inference Strategies, and Ranking Optimization

Recent research extends the application of Pass@k beyond direct evaluation:
- **Inference strategies**: Best-of-Majority (BoM) [2510.03199] selects the $k$ most promising responses from $N$ samples, provably minimizing regret (defined as $1-\mathrm{Pass@}k$) under reward model and coverage constraints.
- **Direct loss optimization**: Top Pass [2408.05715] trains pairwise ranking models to maximize Pass@k by adjusting the margin between the hardest negatives and the best positives, with robust surrogate losses and stabilization via auxiliary classification loss.
- **Advantage shaping**: Policy gradient methods can be unified using surrogate reward functions and regularizers (arcsin, entropy, etc.), allowing tailored emphasis on exploration or exploitation [2510.23049].

A plausible implication is that, for tasks requiring both breadth (covering rare successes) and depth (consistent reasoning), hybrid reporting—Pass@1, Pass@k with $k$ matched to real-world sampling budgets, and Cover@τ at relevant thresholds—provides the most informative assessment.

## References

- "Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries" [2510.08325]
- "Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems" [2505.15201]
- "SimKO: Simple Pass@K Policy Optimization" [2510.14807]
- "Best-of-Majority: Minimax-Optimal Strategy for Pass@$k$ Inference Scaling" [2510.03199]
- "Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models" [2508.10751]
- "Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients" [2510.23049]
- "Don't Pass$\mathtt{@}k$: A Bayesian Framework for Large Language Model Evaluation" [2510.04265]
- "Top Pass: Improve Code Generation by Pass@k-Maximized Code Ranking" [2408.05715]

Source: https://www.emergentmind.com/topics/pass-k-metric-f1419870-1396-4930-8901-987e892e30e3