Pass@k: Evaluating LLM Code and Reasoning

• Pass@k is a metric that measures the probability that at least one of k generated outputs passes a verifier, essential for LLM applications in code and reasoning tasks.
• It drives research into methods like Coordinated Pass@k Policy Optimization, which segments solution generation into planning and solving stages to enhance diversity.
• Analytical trade-offs between pass@k and pass@1 reveal gradient interference issues, leading researchers to adopt blended objectives to balance reliability and diversity.

Pass@$k$ (commonly styled as pass@k) is a fundamental metric and optimization target in evaluating LLMs for verifiable, open-ended tasks such as code generation, mathematical reasoning, and short-answer question answering. The metric quantifies the probability that, out of $k$ independently sampled model outputs for a given input, at least one output passes a programmatic verifier or test suite. Pass@$k$ has shaped both benchmarks and learning objectives in LLM research, giving rise to policy optimization methods explicitly targeting multi-sample verification. The metric, its formalism, empirical observed trade-offs, and algorithmic specialization are central to current LLM evaluation and training regimes (Li et al., 26 May 2026, Barakat et al., 24 Feb 2026).

1. Formal Definition and Analytical Properties

Pass@$k$ is defined with respect to a distribution $\mathcal{D}$ over prompts $x\in\mathcal{X}$ and a stochastic LLM policy $\pi_\theta(y\mid x)$. For each prompt $x$, the single-sample success probability is

$$p_\theta(x) := \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)] \in [0,1],$$

where $r(x, y)\in\{0,1\}$ indicates verifier-confirmed correctness.

For $k$0 i.i.d. samples $k$1, the canonical pass@$k$2 reward is

$k$3

The population-level pass@$k$4 objective is

$k$5

The pass@$k$6 value captures the complementary failure probability: under i.i.d. sampling,

$k$7

If $k$8,

$k$9

The metric is monotonically increasing in $k$0, and $k$1 where $k$2 is concave and increasing.

2. Standard Inference Protocols and Limitations of IID Sampling

The prevailing protocol for code and reasoning task evaluation samples $k$3 continuations per prompt and declares success if any candidate passes the verifier. This approach is computationally straightforward but can be highly sample-inefficient at nontrivial $k$4, especially with modern overconfident LLMs. When the model “collapses” onto a single dominant solution mode, i.i.d. sampling yields numerous near-duplicate outputs—leading to wasted computational budget without true exploration of alternative strategies or algorithms.

This redundancy is particularly detrimental in problem domains—such as competitive programming—where a multiplicity of fundamentally distinct algorithmic ideas may exist, and a single working attempt suffices for pass@$k$5 success (Li et al., 26 May 2026).

3. Coordinated Pass@K Policy Optimization (CPPO) and Algorithmic Advances

To address the inefficiency of i.i.d. sampling, the Coordinated Pass@$k$6 Policy Optimization (CPPO) framework recasts pass@$k$7 solution generation as a joint exploration process over a structured strategy space. The approach factors generation into a two-stage policy:

• Planner $k$8 emits a tuple $k$9 of high-level strategies, autoregressively enforcing novelty across $k$0.
• Shared solver $k$1 generates a complete solution $k$2 from each $k$3.

The resulting trajectory is $k$4, with joint probability

$k$5

Reward assignment incorporates:

• Branch-level verifier rewards $k$6
• Tuple-level outcome reward $k$7, directly matching pass@$k$8
• Plan-validity reward $k$9, gating credit to tuples that are valid (distinct, code-free outlines, minimal topicality)
• Combined planner reward $\mathcal{D}$0

CPPO is trained with Gradient Regularized Proximal Policy Optimization (GRPO) and a split-region advantage scheme: token-level solvers use normalized within-tuple advantages; planners use normalized advantages across sampled tuples.

Empirical results show CPPO consistently outperforms direct sampling and competitive pass@$\mathcal{D}$1-oriented RL baselines across APPS, CodeContests, and LiveCodeBench-v6, achieving statistically significant improvements in six of nine (model, dataset) cells (Li et al., 26 May 2026). As a concrete example, CPPO delivers a gain of $\mathcal{D}$2 (from $\mathcal{D}$3 to $\mathcal{D}$4) on LiveCodeBench-v6 with Qwen3.5-9B compared to PKPO.

4. Analytical Trade-Offs: Pass@K vs Pass@1 and Gradient Interference

Post-training LLMs to maximize pass@$\mathcal{D}$5 has revealed a recurring trade-off: as $\mathcal{D}$6 increases, pass@$\mathcal{D}$7 improves but pass@1 degrades (Barakat et al., 24 Feb 2026). This is operationally significant, given that single-shot inference often remains necessary due to resource or latency constraints.

Analytically, pass@$\mathcal{D}$8 optimization amplifies gradients for “hard” prompts (i.e., those with lower base success probability), via the weighting $\mathcal{D}$9 in the chain-rule expansion of $x\in\mathcal{X}$0. This non-uniform reweighting can result in a global gradient that is anti-aligned with the pass@1 direction, especially in the presence of prompts exhibiting negative interference. For prompt $x\in\mathcal{X}$1, the agreement $x\in\mathcal{X}$2 quantifies its gradient’s alignment with the population mean.

If negatively-interfering, low-success prompts are sufficiently amplified by $x\in\mathcal{X}$3, the overall optimization step for $x\in\mathcal{X}$4 can perversely harm $x\in\mathcal{X}$5, as formalized in Proposition 3.1 and explored empirically. For sufficiently large $x\in\mathcal{X}$6, a phase transition can occur where further increase in $x\in\mathcal{X}$7 guarantees gradient conflict and pass@1 decline.

Empirical validation on mathematical reasoning tasks with LLMs (DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B) confirms that agreement scores stratify by difficulty and that pass@$x\in\mathcal{X}$8 weighting can upweight hard, negatively interfering prompts with power-law disparity—yielding persistent negative gradient inner products and pass@1 regression during pass@$x\in\mathcal{X}$9 optimization (Barakat et al., 24 Feb 2026).

5. Practical Recommendations and Mitigation Strategies

Operational workflows should consider the following:

• When only single-sample inference (pass@1) is feasible, direct pass@$\pi_\theta(y\mid x)$0 optimization without constraint may degrade real-world reliability.
• One can blend the pass@$\pi_\theta(y\mid x)$1 and pass@1 objectives (e.g., interpolate $\pi_\theta(y\mid x)$2 with $\pi_\theta(y\mid x)$3) to regularize training and retain single-sample fidelity.
• Capping extreme values of $\pi_\theta(y\mid x)$4 or employing alternative risk-sensitive objectives (e.g., log-sum-exp over rewards, tuning exponents replacing $\pi_\theta(y\mid x)$5) can smooth the optimization landscape and mitigate outlier prompt amplification.
• Gradient-surgery or prompt-level trust-region methods can be used to remove or limit the influence of strongly negatively interfering prompts.
• Monitoring the gradient inner product $\pi_\theta(y\mid x)$6 can offer a diagnostic signal: if it turns negative, switching to a reduced $\pi_\theta(y\mid x)$7 or incorporating explicit pass@1 regularization is recommended (Barakat et al., 24 Feb 2026).

6. Empirical Benchmarking and Impact

Across major code reasoning benchmarks—APPS, CodeContests, LiveCodeBench-v6—pass@$\pi_\theta(y\mid x)$8 is the primary evaluation metric. The empirical superiority of coordinated policies (as in CPPO) demonstrates that optimizing diversity in attempted solution strategies, rather than naively increasing $\pi_\theta(y\mid x)$9 for i.i.d. model samples, directly translates to higher pass@$x$0—with the incremental benefit most pronounced on harder benchmarks and larger models. Gains of up to $x$1 in pass@$x$2 have been documented for Qwen3.5-4B on LiveCodeBench-v6 compared to direct sampling. This suggests that structured, reward-gated diversification substantially advances the state of the art for verifiable LLM-assisted code synthesis (Li et al., 26 May 2026).

7. Open Challenges and Future Directions

A persistent open problem is maximizing pass@$x$3 without incurring excessive cost to pass@1, especially as LLMs are deployed in settings with heterogeneous inference budgets and risk tolerances. Further research into adaptive, prompt-level trade-offs, refined reward shaping, and population-wide agreement-aware optimizers is ongoing. A plausible implication is that dynamically balancing exploration (diversity) and exploitation (single-shot robustness) will remain central to the next generation of LLM post-training strategies and benchmark design.