Pass@K Evaluation Metric

Updated 28 October 2025

Pass@K is a statistical metric that computes the probability of at least one correct outcome from K independent candidate outputs.
It is widely used to assess models such as large language models, reinforcement learning systems, and program synthesis by emphasizing both quality and diversity.
Optimization techniques for Pass@K include policy gradients and variance reduction methods that balance exploration with reliable performance.

The Pass@K objective is a statistical evaluation and optimization criterion widely used to assess and advance the performance of machine learning systems—especially LLMs, reinforcement learning with verifiable rewards (RLVR), and program synthesis—when multiple candidate outputs per prompt are allowed. Pass@K quantifies the probability that at least one of K independent samples generated by a model yields a correct solution to a given task. This metric is fundamentally distinct from single-attempt (Pass@1) evaluation, as it reflects both the quality and diversity of candidate outputs, aligning evaluation (and, increasingly, optimization) with practical use cases where users or post-processors may select from several generated hypotheses.

1. Formal Definition and Mathematical Foundations

Given a probabilistic model that generates candidate solutions $y_1, ..., y_K$ conditionally independently via its output distribution (for example, $y_i \sim \pi_\theta(\cdot|x)$ ) and a reward function $r(y, a)$ (often binary: correct or not), Pass@K is defined as the probability that at least one of K samples is rewarded:

$\text{Pass@}K = 1 - \prod_{i=1}^K (1 - \mathbb{E}_{y \sim \pi_\theta}[r(y, a)])$

or for notational simplicity,

$\text{Pass@}K(x, a) = 1 - (1 - \rho_\theta(x, a))^K$

where $\rho_\theta(x, a)$ is the probability under the model of generating a correct response to (x, a).

For evaluation over a dataset $D$ , the Pass@K rate is averaged,

$\text{Pass}_p@K = \mathbb{E}_{i \sim D} \left[1 - (1 - p_i)^K\right]$

with $p_i$ the per-task success probability.

This metric is integral in domains where the chance of finding a correct solution increases with the number of independent trials, but perfect reliability per sample is unattainable.

2. Statistical Properties, Limitations, and Alternatives

Pass@K is a nonlinear, monotonically increasing function of the base per-sample success rate $p$ . For any $p > 0$ , $\lim_{K \to \infty} \text{Pass@}K = 1$ , meaning even random guessing will appear successful as the sampling budget grows. This leads to key limitations:

Saturation and Misleading Breadth: At large K, Pass@K may overstate a model's reasoning ability, especially when solutions are “lucky hits” rather than consistently correct (Dragoi et al., 9 Oct 2025).
Variance and Instability: Empirical Pass@K estimates can be highly variable or unstable when the sample size is limited, particularly for moderate or large K (Hariri et al., 5 Oct 2025).
Hidden Diversity Effects: Increased K rewards diversity (exploration), which may not align with robustness or depth of reasoning unless carefully managed (Chen et al., 14 Aug 2025, Peng et al., 16 Oct 2025).

To address these issues, Bayesian evaluation frameworks model the underlying categorical or binary success rates with Dirichlet/Beta priors and employ posterior mean estimation and credible intervals. Compared to Pass@K, this yields more stable ranking, faster convergence, and explicit uncertainty quantification (Hariri et al., 5 Oct 2025).

Alternative “breadth-depth” metrics such as Cover@ $\tau$ (measuring the fraction of tasks for which the per-sample success rate exceeds a fixed $\tau$ ) explicitly separate coverage (breadth) from reliability (depth) (Dragoi et al., 9 Oct 2025). These metrics allow reliable assessment of model capabilities beyond pure hit-rate amplification from repeated sampling.

3. Optimizing for Pass@K: Algorithms, Estimators, and Training Dynamics

Optimization for Pass@K differs markedly from optimizing average (Pass@1) reward. If sampling capacity permits K solutions per prompt, RLVR and LLM systems increasingly tailor their training objectives to maximize Pass@K directly.

Policy Gradient Methods: Two principal strategies are in use:
- Direct REINFORCE-style gradients: Compute unbiased estimators of the gradient of the Pass@K objective, which (for binary rewards) involve multiplying the log-likelihood gradient by a scaling factor reflecting the probability that all K samples fail except the current one (Walder et al., 21 May 2025, Thrampoulidis et al., 27 Oct 2025).
- Advantage Shaping: Modify the per-sample “advantage” in policy optimization (e.g., group relative policy optimization, GRPO) by reweighting samples in proportion to their empirical difficulty or failure rate (“hard-example up-weighting”). This is formally equivalent to maximizing a surrogate (variance-stabilizing) reward function, such as $\arcsin(\sqrt{\text{Pass@}K})$ (Thrampoulidis et al., 27 Oct 2025).
Variance-Reduction Techniques: Leave-one-out baselining (RLOO) and combinatorial aggregation across all subsets of k samples within a mini-batch yield low-variance, unbiased gradient estimators even for small batch sizes (Walder et al., 21 May 2025).
Exploration-Exploitation Tradeoff: Optimization for Pass@1 induces probability mass to concentrate on a few high-confidence outputs, hindering diversity and reducing Pass@K performance for $K > 1$ (Chen et al., 14 Aug 2025, Peng et al., 16 Oct 2025). Conversely, explicit Pass@K optimization balances exploration and exploitation, encouraging higher-entropy distributions and reducing the risk of local optima.
Adaptive/Annealed Objectives: Dynamically varying the target K during training or using composite objectives further enhances both Pass@1 and Pass@K (Walder et al., 21 May 2025, Chen et al., 14 Aug 2025).

Methods such as SimKO (Simple Pass@K Optimization) reinforce this principle by directly addressing probability over-concentration in the output distribution, redistributing gradient signal across top-K candidates for correct outputs and penalizing overconfidence in incorrect outputs, especially for high-entropy tokens (Peng et al., 16 Oct 2025). This approach uniquely tailors the training dynamics at the token level.

4. Inference-Time Algorithms and Scaling Laws

Pass@K not only defines training objectives but also motivates specialized inference and selection algorithms:

Best-of-N (BoN) and Majority Voting: Naively select the top candidate(s) or choose by empirical frequency. However, these methods are suboptimal regarding regret scaling and can perform worse as the sample size N increases if the reward model overoptimizes (Di et al., 3 Oct 2025).
Best-of-Majority (BoM): Combines frequency filtering (to retain high-likelihood candidates) with reward-based selection, yielding minimax-optimal regret bounds as a function of $k, N$ , model capacity, and reward model accuracy:

$\text{Regret}(x) \le \epsilon_\mathrm{opt}(x) + O\left(\sqrt{\frac{\epsilon_\mathrm{RM}^2(x)\, C^*}{k}}\right)$

where $C^*$ is the coverage coefficient (inverse probability of hitting the optimal answer) (Di et al., 3 Oct 2025).
Scaling Law Prediction: Efficient estimation of Pass@K at large K from limited data is crucial for model risk assessment and deployment. Conventional scaling law approaches (log-log regression, discretized beta fitting) are statistically flawed—assumptions of independence and noise structure are violated (Kazdan et al., 6 Oct 2025). A robust solution employs beta-binomial modeling (parameter inference via MLE) for the per-problem success probability and computes Pass@K as

$\text{Pass@}K = \mathbb{E}_{p \sim \mathrm{Beta}(\alpha, \beta)}[1-(1-p)^K]$

Adaptive sampling strategies further improve estimator efficiency by concentrating additional samples on the hardest problems (Kazdan et al., 6 Oct 2025).

5. Robust System Design and Practical Applications

Pass@K-oriented methodologies are widely implemented across several domains:

LLM Evaluation: Capabilities (as well as safety risks such as jailbreak susceptibility) are often more apparent under repeated sampling. Regulatory processes and scientific benchmarking including Pass@K forecast use robust statistical estimation to quantify their extrapolation to large user populations (Kazdan et al., 6 Oct 2025).
Code Generation and Program Synthesis: Strategic candidate ranking, optimized for Pass@K, improves the probability that correct code is among the top K candidates surfaced to the user, aligning ranking loss functions directly with Pass@K (Lyu et al., 11 Aug 2024).
Reinforcement Learning with Verifiable Rewards: Algorithms like PKPO and SimKO employ reward transformations and gradient estimators designed explicitly for Pass@K improvement, yielding more robust policy exploration and success on hard tasks (Walder et al., 21 May 2025, Peng et al., 16 Oct 2025).
Task-Agnostic Performance Enhancement: Techniques that leverage intrinsic model inconsistency (such as the “Variator” approach) diversify response pathways and thereby amplify Pass@K by generating semantically distinct variants per request, especially for hard problems (Dalal et al., 19 May 2025).
Quantum Machine Learning and Control: By ensuring neural network ansätze converge to globally optimal geodesics (e.g., via Cartan KAK decompositions), quantum control tasks can improve the probability that at least one candidate achieves the desired objective (analogous to Pass@K performance) (Perrier, 2 Apr 2025).

Application-specific trade-offs, such as the required reliability per task and available computational budget, guide the desired choice of K and influence whether more robust “depth” metrics (Cover@ $\tau$ ) or uncertainty-aware Bayesian evaluation frameworks are also adopted.

6. Theoretical Interpretations and Emerging Directions

Surrogate Reward Perspective: Advantage shaping and gradient reweighting for Pass@K are special cases of maximizing variance-stabilized (arcsin-transformed) or otherwise regularized surrogate reward objectives. Hard-example up-weighting is interpretable as reward-level regularization, directly connecting empirical heuristic modifications to a rigorous statistical basis (Thrampoulidis et al., 27 Oct 2025).
Bias, Variance, and Hardness Sensitivity: Pass@K amplifies small improvements on difficult cases, especially as K increases, making robust estimation and regularization critical. Gradient estimators and transformation functions are crafted for stability and analytic tractability.
Beyond Pass@K: Breadth-Depth and Bayesian Metrics: To prevent misinterpretation of capability boundaries, alternative metrics such as Cover@ $\tau$ (fraction of problems solved with at least $\tau$ reliability) and directly Bayesian posterior mean estimates (with credible intervals) are increasingly recommended over or in addition to raw Pass@K (Dragoi et al., 9 Oct 2025, Hariri et al., 5 Oct 2025).

In summary, the Pass@K objective is a foundational metric for evaluating and optimizing the performance of machine learning systems when repeated sampling is permitted. Its adoption drives innovations in both algorithmic training and inference design, spurs the development of robust statistical estimators for large-scale assessment, and motivates complementary metrics for reliability and depth. As machine learning systems become more widely deployed and scrutinized, Pass@K and its analytical framework provide essential tools for both capability advancement and rigorous evaluation.