---
title: 'Pass@k: Evaluating LLM Code and Reasoning'
url: https://www.emergentmind.com/topics/pass-o
type: topic
---

# Pass@k: Evaluating LLM Code and Reasoning

Pass@$k$ (commonly styled as pass@k) is a fundamental metric and optimization target in evaluating large language models (LLMs) for verifiable, open-ended tasks such as code generation, mathematical reasoning, and short-answer question answering. The metric quantifies the probability that, out of $k$ independently sampled model outputs for a given input, at least one output passes a programmatic verifier or test suite. Pass@$k$ has shaped both benchmarks and learning objectives in LLM research, giving rise to policy optimization methods explicitly targeting multi-sample verification. The metric, its formalism, empirical observed trade-offs, and algorithmic specialization are central to current LLM evaluation and training regimes [2605.27000][2602.21189].

## 1. Formal Definition and Analytical Properties

Pass@$k$ is defined with respect to a distribution $\mathcal{D}$ over prompts $x\in\mathcal{X}$ and a stochastic LLM policy $\pi_\theta(y\mid x)$. For each prompt $x$, the single-sample success probability is 
$$
p_\theta(x) := \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)] \in [0,1],
$$
where $r(x, y)\in\{0,1\}$ indicates verifier-confirmed correctness.

For $k$ i.i.d. samples $y_1,\ldots,y_k\sim\pi_\theta(\cdot\mid x)$, the canonical pass@$k$ reward is
$$
R_{\mathrm{pass@}k}(x, y_{1:k}) = \mathbf{1}\left\{\max_{i=1}^k V(x, y_i) = 1\right\}.
$$
The population-level pass@$k$ objective is
$$
J_k(\theta) := \mathbb{E}_{x\sim\mathcal{D}}\Big[1-(1-p_\theta(x))^k\Big].
$$

The pass@$k$ value captures the complementary failure probability: under i.i.d. sampling, 
$$
\Pr\left[R_{\mathrm{pass@}k}=1\right] = 1-\prod_{i=1}^k(1-p_i).
$$
If $p_i\equiv p$, 
$$
\operatorname{pass}@k = 1 - (1-p)^k.
$$

The metric is monotonically increasing in $k$, and $J_1(\theta)\leq J_k(\theta)\leq \min(1, f_k(J_1(\theta)))$ where $f_k(p) = 1-(1-p)^k$ is concave and increasing.

## 2. Standard Inference Protocols and Limitations of IID Sampling

The prevailing protocol for code and reasoning task evaluation samples $k$ continuations per prompt and declares success if any candidate passes the verifier. This approach is computationally straightforward but can be highly sample-inefficient at nontrivial $k$, especially with modern overconfident LLMs. When the model “collapses” onto a single dominant solution mode, i.i.d. sampling yields numerous near-duplicate outputs—leading to wasted computational budget without true exploration of alternative strategies or algorithms.

This redundancy is particularly detrimental in problem domains—such as competitive programming—where a multiplicity of fundamentally distinct algorithmic ideas may exist, and a single working attempt suffices for pass@$k$ success [2605.27000].

## 3. Coordinated Pass@K Policy Optimization (CPPO) and Algorithmic Advances

To address the inefficiency of i.i.d. sampling, the Coordinated Pass@$K$ Policy Optimization (CPPO) framework recasts pass@$k$ solution generation as a joint exploration process over a structured strategy space. The approach factors generation into a two-stage policy:

- **Planner** $q_\Theta$ emits a tuple $S = (s_1,\ldots,s_K)$ of high-level strategies, autoregressively enforcing novelty across $s_i$.
- **Shared solver** $p_\Theta$ generates a complete solution $y_i$ from each $s_i$.

The resulting trajectory is $\tau=(S, y_{1:K})$, with joint probability
$$
\pi_\Theta(\tau \mid x) = q_\Theta(S \mid x)\prod_{i=1}^K p_\Theta(y_i \mid x, s_i).
$$

Reward assignment incorporates:
- **Branch-level verifier rewards** $(r^{\mathrm{out}}_i = V(x, y_i))$
- **Tuple-level outcome reward** $R_{\mathrm{out}}(x, y_{1:K}) = \mathbf{1}\{\max_i r^{\mathrm{out}}_i = 1\}$, directly matching pass@$k$
- **Plan-validity reward** $J_\psi(x,S)\in\{0,1\}$, gating credit to tuples that are valid (distinct, code-free outlines, minimal topicality)
- **Combined planner reward** $R_{\mathrm{plan}}(x,S,y_{1:K}) = J_\psi(x,S)\times R_{\mathrm{out}}(x, y_{1:K})$

CPPO is trained with Gradient Regularized Proximal Policy Optimization (GRPO) and a split-region advantage scheme: token-level solvers use normalized within-tuple advantages; planners use normalized advantages across sampled tuples.

Empirical results show CPPO consistently outperforms direct sampling and competitive pass@$k$-oriented RL baselines across APPS, CodeContests, and LiveCodeBench-v6, achieving statistically significant improvements in six of nine (model, dataset) cells [2605.27000]. As a concrete example, CPPO delivers a gain of $+0.160$ (from $0.588$ to $0.748$) on LiveCodeBench-v6 with Qwen3.5-9B compared to PKPO.

## 4. Analytical Trade-Offs: Pass@K vs Pass@1 and Gradient Interference

Post-training LLMs to maximize pass@$k$ has revealed a recurring trade-off: as $k$ increases, pass@$k$ improves but pass@1 degrades [2602.21189]. This is operationally significant, given that single-shot inference often remains necessary due to resource or latency constraints.

Analytically, pass@$k$ optimization amplifies gradients for “hard” prompts (i.e., those with lower base success probability), via the weighting $w_k(p)=k(1-p)^{k-1}$ in the chain-rule expansion of $\nabla J_k$. This non-uniform reweighting can result in a global gradient that is anti-aligned with the pass@1 direction, especially in the presence of prompts exhibiting *negative interference*. For prompt $x$, the agreement $a_\theta(x) = \langle\nabla p_\theta(x), \nabla J_1(\theta)\rangle$ quantifies its gradient’s alignment with the population mean. 

If negatively-interfering, low-success prompts are sufficiently amplified by $w_k(p)$, the overall optimization step for $J_k$ can perversely harm $J_1$, as formalized in Proposition 3.1 and explored empirically. For sufficiently large $k$, a phase transition can occur where further increase in $k$ guarantees gradient conflict and pass@1 decline.

Empirical validation on mathematical reasoning tasks with LLMs (DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B) confirms that agreement scores stratify by difficulty and that pass@$k$ weighting can upweight hard, negatively interfering prompts with power-law disparity—yielding persistent negative gradient inner products and pass@1 regression during pass@$k$ optimization [2602.21189].

## 5. Practical Recommendations and Mitigation Strategies

Operational workflows should consider the following:

- When only single-sample inference (pass@1) is feasible, direct pass@$k$ optimization without constraint may degrade real-world reliability.
- One can blend the pass@$k$ and pass@1 objectives (e.g., interpolate $J_k$ with $J_1$) to regularize training and retain single-sample fidelity.
- Capping extreme values of $w_k$ or employing alternative risk-sensitive objectives (e.g., log-sum-exp over rewards, tuning exponents replacing $k$) can smooth the optimization landscape and mitigate outlier prompt amplification.
- Gradient-surgery or prompt-level trust-region methods can be used to remove or limit the influence of strongly negatively interfering prompts.
- Monitoring the gradient inner product $\langle \nabla J_k, \nabla J_1 \rangle$ can offer a diagnostic signal: if it turns negative, switching to a reduced $k$ or incorporating explicit pass@1 regularization is recommended [2602.21189].

## 6. Empirical Benchmarking and Impact

Across major code reasoning benchmarks—APPS, CodeContests, LiveCodeBench-v6—pass@$k$ is the primary evaluation metric. The empirical superiority of coordinated policies (as in CPPO) demonstrates that optimizing diversity in attempted solution strategies, rather than naively increasing $k$ for i.i.d. model samples, directly translates to higher pass@$k$—with the incremental benefit most pronounced on harder benchmarks and larger models. Gains of up to $+0.321$ in pass@$4$ have been documented for Qwen3.5-4B on LiveCodeBench-v6 compared to direct sampling. This suggests that structured, reward-gated diversification substantially advances the state of the art for verifiable LLM-assisted code synthesis [2605.27000].

## 7. Open Challenges and Future Directions

A persistent open problem is maximizing pass@$k$ without incurring excessive cost to pass@1, especially as LLMs are deployed in settings with heterogeneous inference budgets and risk tolerances. Further research into adaptive, prompt-level trade-offs, refined reward shaping, and population-wide agreement-aware optimizers is ongoing. A plausible implication is that dynamically balancing exploration (diversity) and exploitation (single-shot robustness) will remain central to the next generation of LLM post-training strategies and benchmark design.

Source: https://www.emergentmind.com/topics/pass-o