---
title: Pass@k Optimization-Induced Pass@1 Degradation
url: https://www.emergentmind.com/topics/pass-k-optimization-induced-pass-1-degradation
type: topic
---

# Pass@k Optimization-Induced Pass@1 Degradation

Pass@k optimization-induced Pass@1 degradation refers to the empirically and theoretically validated phenomenon that arises when reinforcement learning with verifiable rewards (RLVR) directly optimizes the pass@k metric—defined as the probability that at least one of k independent samples from a policy is correct—rather than the more conventional pass@1 (single-sample) success rate. While intuitively appealing for encouraging exploration, policy-gradient ascent on pass@k can, under typical conditions, reduce or stall the single-trial accuracy (pass@1), especially for tasks featuring discrete answer spaces, hard/easy prompt interference, or policy entropy collapse. This trade-off has been rigorously formulated in recent literature, with studies elucidating the underlying gradient dynamics, empirical manifestations on language models, and strategies for mitigation [2511.16231][2510.08325][2508.10751][2602.21189][2505.15201][2510.23049][2508.14029].

## 1. Theoretical Foundations: Pass@k and Its Policy Gradient

Given a prompt $x$ and an RL policy $\pi_\theta(y|x)$, pass@1 is $J_1(x;\theta) = \mathbb{P}_{y\sim\pi_\theta}[V(x,y)=1]$. Pass@k generalizes this to $J_k(x;\theta) = 1 - (1 - J_1(x;\theta))^k$, the probability that at least one of $k$ independent samples is correct [2511.16231][2602.21189]. The joint policy gradient is given by
\[
\nabla_\theta J_k(x;\theta) = k\,(1-J_1(x;\theta))^{k-1}\;\nabla_\theta J_1(x;\theta)
\]
[2511.16231][2602.21189][2505.15201][2510.23049].
This implies that optimizing pass@k is a nonnegative, per-instance reweighting of the pass@1 gradient, with the scalar factor $\alpha_k(x;\theta) = k(1-J_1(x;\theta))^{k-1}$. As such, the direction of optimization does not change, but the allocation of gradient magnitude shifts toward instances with low single-sample success (i.e., hard prompts).

## 2. Mechanisms and Regimes of Degradation

### Vanishing Learning Signal and Exploration Collapse

Two principal pathologies emerge when optimizing pass@k [2511.16231][2510.08325]:

- **Low-success regime**: For instances where $J_1(x;\theta) \approx 0$, $\alpha_k$ is large but the empirical REINFORCE gradient is virtually always zero because correct samples are rarely observed. Thus, even with high reweighting, the signal vanishes.
- **High-success regime**: As $J_1(x;\theta) \to 1$, $\alpha_k \to 0$, leading to near-zero gradients for problems the model already solves well. The policy ceases to fine-tune confident outputs, which impedes further improvement in pass@1.

When the policy collapses onto a single solution mode (exploration collapse), both pass@1 and pass@k saturate, and the gap $\Delta(k) = \text{pass@k} - \text{pass@1}$ shrinks to zero. In practice, this often means the model over-commits to the earliest discovered mode, potentially missing better optima and locking out improved solutions [2511.16231].

### Prompt Interference and Gradient Conflict

Recent work introduces the concept of "prompt interference" to formalize how optimizing pass@k may actively decrease pass@1 [2602.21189]. The key object is the prompt-interference kernel $\kappa_\theta(x,x') = \langle \nabla p_\theta(x),\nabla p_\theta(x')\rangle$. Prompts for which $a_\theta(x):= \langle \nabla J_1(x;\theta),\nabla J_1(\theta)\rangle < 0$ are termed negatively interfering: improving performance on such prompts can directly reduce the average pass@1. Pass@k policy gradients disproportionately emphasize these hard, negatively interfering samples, and if their aggregated covariance is sufficiently large and negative, the overall step in parameter space provably rotates away from the pass@1 direction, decreasing single-shot accuracy even as pass@k increases (Proposition 3.1 and 3.4 in [2602.21189]).

## 3. Empirical Manifestations in RLVR and LLM Fine-tuning

Empirical studies confirm the theoretical findings across mathematical reasoning, code synthesis, and traversal tasks [2511.16231][2510.08325][2508.10751][2505.15201][2602.21189]:

- **Crossover Phenomenon**: RLVR models fine-tuned for pass@k often yield higher performance at small k (e.g., pass@1, pass@4), but for large k, pretrained base models surpass RLVR models in pass@k, and RLVR models may demonstrate lower pass@1 on held-out tasks [2510.08325].
- **Entropy Collapse**: RLVR geared toward pass@1 drives policy entropy downward, leading to limited sample diversity and stalling of pass@k at large k [2508.14029].
- **Gradient Allocation Bias**: Optimization on pass@k with large k specifically targets increasing the coverage (breadth) of low-success prompts (lifting $p_i$ from near 0 to near 0.1), sacrificing the refinement (depth) of tasks with $p_i$ near 1, negatively impacting pass@1 [2510.08325][2508.10751].
- **Negative Covariance Confirmation**: Quantitative MATH dataset experiments demonstrate that the inner product between pass@k and pass@1 gradients can become negative when pass@k's prompt weighting aligns with negatively interfering prompts [2602.21189].

## 4. Analytical and Algorithmic Explanations

### Gradient and Advantage Structures

Pass@k policy gradients, both in REINFORCE and advantage-shaping schemes (GRPO, RLOO), can be interpreted as mixtures of per-sample gradients, with sharp reweighting toward hard examples and vanishing signals for easy ones [2510.23049][2508.10751][2505.15201]. For instance, the sum-of-absolute-advantage $\eta(N_\text{pos})$ peaks at intermediate accuracy under pass@k and rapidly decays in the high-accuracy regime, sharply down-weighting further improvements in high-confidence cases. Direct pass@k training therefore "kills" the gradient on easy examples, entrenching mediocrity in pass@1 [2508.10751][2510.23049].

### Impact of Discrete Answer Spaces

In domains with small, discrete answer spaces (e.g., numeric math), pass@k optimization at large k is especially prone to misleading the practitioner: any nonzero probability mass ('random guessing') is, by combinatorial accumulation, sufficient for high pass@k, even if $p_i$ is tiny. This degeneracy results in models that appear highly capable at large sample budgets, yet have unreliable or even degraded single-shot accuracy [2510.08325][2511.16231].

## 5. Mitigation Strategies and Advanced Techniques

To counteract pass@k-induced pass@1 degradation, a suite of strategies has been proposed and validated:

- **Explicit Exploration Bonuses**: Incorporate entropy regularization or maximum-coverage rewards to maintain exploration and avoid premature mode collapse [2511.16231][2508.14029].
- **Multi-objective Training**: Jointly optimize convex combinations of pass@1 and pass@k or use risk-sensitive objectives to balance breadth and depth [2602.21189][2508.10751][2505.15201].
- **Advantage Shaping and Blending**: Apply combination advantage functions, blend per-example gradients by current accuracy or entropy, or use smoothed surrogate rewards to preserve useful learning signals on both hard and easy prompts [2510.23049][2508.10751].
- **Tempered Reweighting**: Replace k in the weighting function with a tunable γ or anneal k during training, starting from k>1 for exploration and reducing to k=1 for exploitation [2505.15201][2508.10751][2511.16231].
- **Gradient Surgery and Prompt Curation**: Project out negatively interfering gradient directions or train subsets of prompts with more conservative objectives [2602.21189].
- **Online Data Diversification (SvS)**: Use self-play and variational problem synthesis to continually synthesize challenging variants, thereby promoting entropy and sustaining both pass@k and pass@1 [2508.14029].

## 6. Diagnostic and Evaluation Best Practices

A consensus emerges that pass@k, although a valuable diagnostic of latent coverage, is unsuitable as a sole training objective [2511.16231][2510.08325]. For evaluation and reporting:

- Always supplement pass@k with single-trial metrics (pass@1) and coverage curves (Cover@τ at moderate-to-high τ) to diagnose reliability and genuine reasoning ability [2510.08325].
- Be cautious in interpreting high pass@k at large k, especially in finite-support or small-answer-space tasks, as this may reflect random guessing.
- Employ coverage-area-under-the-curve metrics (e.g., AvgAUC^+_cover) for aggregate, pairwise-aware model ranking [2510.08325].

### Table: Summary of Key Mitigation Techniques

| Technique                   | Principle                        | Effect on Pass@1/Pass@k    |
|-----------------------------|----------------------------------|---------------------------|
| Entropy regularization      | Promotes exploration             | Preserves both            |
| Multi-objective blending    | Trades off coverage/accuracy     | Balances                  |
| Annealing k                 | Early exploration, final exploit | Boosts both if scheduled  |
| Advantage shaping           | Smooths reward/gradient decay    | Avoids vanishing signal   |
| Online data synthesis (SvS) | Sustains problem and solution diversity | Extends both boundaries    |

## 7. Outlook and Open Challenges

The optimization-induced pass@1 degradation when training directly on pass@k reflects a broader challenge in reinforcement learning for reasoning: aligning exploration incentives with policy reliability. Analytical findings establish the inevitability of vanishing gradients and counterproductive gradient directions in naive pass@k policy optimization, especially in the presence of prompt interference and low-entropy policies [2511.16231][2602.21189]. Modern RLVR approaches appear to benefit from strategically decoupling metric evaluation (diagnostics at multiple k and reliability thresholds) from training objectives, blending exploration-promoting and accuracy-consolidating methodologies, and continually expanding the frontier of both solution depth and breadth [2508.14029][2505.15201][2510.08325].

Active research directions include more precise characterization of gradient conflicts across task distributions, development of robust surrogate rewards and dynamic curricula, and systematic benchmarking of adaptive schedules or entropy-aware advantage shaping schemes in large-scale LLM post-training.

Source: https://www.emergentmind.com/topics/pass-k-optimization-induced-pass-1-degradation