---
title: Pass@$k$ Metric in Code Generation & RL
url: https://www.emergentmind.com/topics/pass-k-metric-b5b58688-14e3-4ed9-b1f7-504db4b60803
type: topic
---

# Pass@$k$ Metric in Code Generation & RL

The Pass@$k$ metric is a probabilistic evaluation measure widely adopted in code generation, reasoning, and reinforcement learning tasks to quantify the ability of a model or inference strategy to produce at least one correct solution within $k$ independently sampled outputs. Conceptually, Pass@$k$ measures the likelihood that a correct response appears within the top $k$ candidates, forming the basis for both evaluation and, increasingly, model optimization protocols in applications where users or downstream systems are permitted to select from a pool of alternatives rather than a single solution.

## 1. Formal Definition and Utility

Let $f(x)$ be a binary indicator for the correctness of model output $x$ (with $f(x) = 1$ if correct, $0$ otherwise), and suppose that a model generates $k$ independent outputs $x_1, \dots, x_k$ for a given input or prompt. The Pass@$k$ metric is defined as:
$$
\text{Pass@}k = \mathbb{E}\left[1 - \prod_{i=1}^k (1 - f(x_i))\right]
$$
This expresses the expected probability that at least one of the $k$ samples is correct. In practical systems—such as large language models (LLMs) deployed for code synthesis or open-ended problem solving—Pass@$k$ is particularly salient because it aligns with real-world practices where users can inspect or utilize multiple generated candidates [2505.15201].

Beyond evaluation, Pass@$k$ is increasingly used as an explicit target for optimization in reinforcement learning (RL) and ranking scenarios, directly shaping how models are trained to allocate probability mass over output spaces to maximize user- or application-centric success rates [2408.05715, 2508.10751].

## 2. Optimization Techniques and Algorithmic Considerations

Traditional models optimized to maximize Pass@$1$ (single-sample correctness) often underutilize the benefit of batch generation and are prone to overly conservative exploitation strategies. Modern optimization schemes incorporate Pass@$k$ directly into their objectives, either through surrogates in ranking losses [2408.05715], reward transformations in RL [2505.15201, 2508.10751], or custom advantage functions. Key methodological developments include:

- **Direct Metric Optimization:** Methods such as Top Pass reformulate the model's loss to directly reflect the Pass@$k$ objective. For a candidate set with positives $C_+$ and negatives $C_-$, ranking is cast as ensuring the highest-scoring positive is above the $k$-th negative:
  $$
  \text{pass@}k = \mathbb{I}[f(Q, C_{+,1}) > f(Q, C_{-,k})]
  $$
  where $f(Q, C)$ is the ranker's score [2408.05715].

- **Surrogate and Analytical Loss Functions:** The non-differentiability of indicator-based Pass@$k$ loss is addressed by adopting surrogates such as squared hinge loss, enabling gradient-based optimization. Analytical derivations in RL permit the closed-form computation of advantage functions leveraging group statistics over $k$ samples, reducing the variance relative to sampling- or bootstrap-based estimators [2508.10751].

- **Reward Transformations in RL:** Pass-at-k Policy Optimization (PKPO) introduces unbiased, low-variance estimators for both binary and continuous reward settings. PKPO generalizes earlier work restricted to $k = n$ and enables annealing of $k$ during training, which empirically improves both Pass@$1$ and Pass@$k$ when training large reasoning models [2505.15201, 2508.10751].

- **Adaptive Grouping:** By structuring rollouts in groups of $k$ and calculating group rewards via max operations, models are explicitly incentivized to explore a diverse output space [2508.10751].

## 3. Inference-Time Strategies for Pass@$k$

Practical inference in Pass@$k$ regimes may involve selecting up to $k$ responses from a larger batch of $N$ candidates. Several strategies have been proposed and analyzed for this selection step, including:

| Strategy                 | Principle                       | Limitations                                   |
|--------------------------|---------------------------------|-----------------------------------------------|
| Majority Voting          | Selects most-frequent outputs   | Constant regret; does not improve with $N$    |
| Best-of-N (BoN)          | Picks $k$ top-ranked by reward  | Susceptible to reward model overoptimization  |
| Best-of-Majority (BoM)   | Filters by empirical frequency, then selects $k$ via reward | Minimax-optimal and scaling-monotonic, robust to reward model errors [2510.03199] |

BoM, in particular, achieves a regret bound of $O(\epsilon_{\text{opt}} + \sqrt{\epsilon^2_{\mathrm{RM}} C^* / k})$ (where $C^*$ is the reference policy’s coverage coefficient), matching theoretical lower bounds and providing robustness as the number of generated candidates $N$ increases [2510.03199].

## 4. Empirical and Theoretical Insights

Pass@$k$-centric approaches have demonstrated strong empirical improvements across multiple domains:

- **Code Generation:** Pass@$k$-maximizing rankers achieve significant gains in top-ranked prediction accuracy. For example, on CodeContests, Top Pass improved pass@$1$ by 32.9% relative to strong baselines, with similar trends on APPS, MBPP, and HumanEval [2408.05715].

- **Reinforcement Learning Tasks:** PKPO and analytical Pass@$k$ training have unblocked learning on problems where Pass@$1$-optimized policies stall, due to improved exploration and better utilization of the candidate pool [2505.15201, 2508.10751].

- **Inference-Scaling:** Theoretical analyses prove that, under proper inference strategies (e.g., BoM), the error scales optimally as $1/\sqrt{k}$ with respect to $k$ and benefits from increases in sampling budget $N$ [2510.03199]. BoN and majority voting do not generally exhibit this monotonic scaling.

- **Iterative Settings:** In agentic or refinement-based systems, the simple Pass@$k$ metric may fail to reflect efficiency because a high pass@$1$ might be achieved via excessive refinements. To address this, Pass@ARC combines Pass@$k$ with a penalty for refinement steps, e.g.,
  $$
  \text{Pass}@\mathrm{ARC} = \text{PassRate} \times \exp(-0.01 (\mathrm{ARC} - 1)^2)
  $$
  [2503.16514].

## 5. Exploration, Exploitation, and Advantage Design

One of the principal findings across recent work is that Pass@$k$-based optimization can simultaneously promote exploration (diversity in sampled solutions) and exploitation (high-confidence, correct outputs). Analytical studies of the advantage function in RLVR show that the sum of optimization “strength” shifts toward harder problems as $k$ increases, biasing training towards scenarios with limited early success. Adaptive advantage modification based on problem difficulty or entropy further fine-tunes this balance, enabling models to focus on challenging instances without sacrificing accuracy on easier cases [2508.10751].

## 6. Limitations, Variants, and Practical Implications

While Pass@$k$ is robust as an evaluation metric where multiple attempts are permissible, it assumes independence of samples and may not be fully informative for settings where the notion of sample diversity or refinement steps is prominent. Extensions such as Pass@ARC address such shortcomings by penalizing inefficiency.

For practitioners, Pass@$k$-aligned methods improve user experience by ensuring that the chance of encountering a usable or correct output within a small number of candidate generations is maximized—directly correlating with reduced manual examination and verification effort in applications ranging from code generation to automated system synthesis [2408.05715, 2503.16514].

## 7. Summary Table of Metric Variants and Strategies

| Metric/Strategy      | Definition/Mechanism                                      | Notable Strengths                |
|---------------------|-----------------------------------------------------------|----------------------------------|
| Pass@$k$            | Probability at least one of $k$ outputs is correct        | Aligns with user experience      |
| Pass@ARC            | Penalizes excess refinement cycles in success rate        | Captures efficiency of solutions |
| BoM                 | Filters by frequency, selects by reward                   | Minimax-optimal, robust scaling  |
| PKPO & Analytical   | Optimizes joint sample utility for Pass@$k$                | Low-variance, effective for RL   |

In conclusion, the Pass@$k$ metric and its variants have become central to the evaluation and optimization of systems in code generation, reasoning, and reinforcement learning, aligning both experimental and theoretical progress with the realities of user-facing performance in multi-candidate settings [2408.05715, 2503.16514, 2505.15201, 2508.10751, 2510.03199].

Source: https://www.emergentmind.com/topics/pass-k-metric-b5b58688-14e3-4ed9-b1f7-504db4b60803