Pass@$o$ Metric in RLVR

• Pass@$o$ Metric is a group-based evaluation tool that measures the probability of producing at least one correct answer among k attempts.
• It integrates exploration and exploitation by using maximal group rewards to promote diverse reasoning and prevent policy collapse.
• Its analytical derivation reduces variance in advantage estimates, enabling more efficient and stable training in large reasoning models.

The Pass@$o$ metric, frequently denoted as Pass@k in recent literature, is a group-based evaluation and training metric for @@@@1@@@@, particularly LLMs within reinforcement learning with verifiable rewards (RLVR). It measures the probability that at least one positively rewarded (i.e., correct) response is produced among $k$ independently generated attempts for a given input. Pass@k not only serves as an evaluative yardstick but also functions as a reward shaping mechanism, rebalancing the classic exploration-exploitation trade-off and offering analytical tractability that supports stable and efficient training dynamics. Its practical implementation enables more adaptive learning, stronger generalization, and improved efficiency in escaping local optima.

1. Formal Definition and Role

The Pass@k metric is defined for a given input $x$ and reference output $y$, where $k$ model responses $\{\hat{y}_i\}_{i=1}^k$ are sampled independently according to policy $\pi_\theta(\cdot|x)$. A verifier assesses each response and assigns a binary reward, $R_i = 1$ if $\hat{y}_i$ is correct according to the task specification, and $R_i = 0$ otherwise. Pass@k is given formally by: $$\text{Pass@k} = \mathbb{E}_{(x,y)\sim D, \{\hat{y}_i\}_{i=1}^k\sim\pi_\theta(\cdot|x)}\left[ \max(R_1, \ldots, R_k) \right]$$ Unlike Pass@1, which assigns reward based exclusively on a single rollout, Pass@k aggregates knowledge over $k$ samples and rewards the maximal response in the group. In training, the maximum reward signal is used to reinforce the probability of producing diverse outputs that increase the chance of hitting at least one correct answer, thereby integrating exploration directly into the reward structure.

2. Pass@k in RLVR: Exploration and Exploitation

Balancing exploration (novel, diverse responses) and exploitation (reinforcing existing high-reward patterns) is a central challenge in RLVR. Pass@k inherently encourages exploration by providing reward if any response in a group is correct. This incites the model to diversify its reasoning strategies, reducing premature convergence to conservative policies often seen in Pass@1 training.

Empirical findings demonstrate that Pass@k training maintains higher answer diversity among negative responses, preventing collapse to a single generic failure mode. Furthermore, policy entropy remains elevated, substantiating sustained exploration. The sum of absolute advantage, $$\eta = N_{pos} \cdot |\hat{A}_{pos}| + N_{neg} \cdot |\hat{A}_{neg}|$$, quantifies aggregate optimization strength and shifts under Pass@k toward improving harder cases—observed as a peak near 25% correctness versus a symmetric focus at 50% with Pass@1.

Training Regime	Exploration Incentive	Exploitation Bias
Pass@1	Low	High
Pass@k	High	Balanced

This structure clarifies how Pass@k counteracts the tendency of single-response reward schemes to over-exploit, facilitating improved coverage of the task’s solution space.

3. Analytical Derivation of Pass@k Advantage

Early implementations relied on sampling and bootstrap aggregation, which are computationally intensive and introduce variance in the estimation of policy advantages. The analytical solution derived for Pass@k advantage enables precise calculation and accelerates training:

• Group reward average:

$$\overline{R}^{group} = 1 - \frac{\binom{N_{neg}}{k}}{\binom{N_{rollout}}{k}}$$

This quantifies the probability that at least one out of $k$ sampled responses is correct.

• Group reward standard deviation:

$$\sigma^{group} = \sqrt{\overline{R}^{group} (1 - \overline{R}^{group})}$$

• Response-relative advantage: Given via assignment from group to individual responses:
  • For positive responses:

  $$\hat{A}_{pos} = \frac{1 - \overline{R}^{group}}{\sigma^{group}}$$ - For negative responses:

  $$\hat{A}_{neg} = \left( 1 - \overline{R}^{group} - \frac{\binom{N_{neg}-1}{k-1}}{\binom{N_{rollout}-1}{k-1}} \right) \cdot (\sigma^{group})^{-1}$$

The analytical formulation streamlines computation, reduces variance, and makes possible adaptive advantage design with group-based metrics, supporting more robust training in high-dimensional policy spaces.

4. Advantage Function Design

The advantage function in Pass@k training is computed on grouped samples rather than individual responses, contextualizing the advantage by the probability of group success. Each response in a group, regardless of outcome, receives the group’s advantage—differentiated by group structure (all-negative or containing positives). The net effect is to shift the optimization strength toward harder problems with lower correctness rates, as shown empirically in the paper.

Adaptations—including hybrids that scale or combine Pass@1 and Pass@k advantages—allow researchers to implicitly focus optimization on unsolved or challenging instances while continuing to benefit from exploitation where appropriate. This marks a transition from explicit reward signal engineering toward implicit advantage-based reward design, enabling more nuanced and dynamic control over learning dynamics.

5. Empirical Outcomes and Generalization

The application of Pass@k as a reward in training yields several robust benefits:

• Downstream Pass@k evaluative metrics improve steadily through training with Pass@k reward without degrading Pass@1 performance.
• Analytical derivation reduces variance in advantage estimates and enhances training stability, particularly in long-horizon settings.
• Models trained with Pass@k demonstrate greater answer diversity, elevated policy entropy, and increased efficiency in escaping local optima.
• Sequential training strategies—such as initial Pass@k training followed by Pass@1 fine-tuning—enable models of modest parameter count to exceed performance of larger, purely exploitation-trained baselines.
• Cross-domain experimental results reinforce the method’s generalizability and its ability to adapt to varying difficulty distributions.

A plausible implication is that metrics and intermediate training objectives derived from Pass@k may offer routes to more scalable and effective reinforcement learning algorithms in the context of complex reasoning tasks.

6. Implications for RLVR Paradigm

Pass@k metric reconfigures the role of exploration in RLVR: instead of being a residual effect of global optimization strategies, exploration is entwined with reward and advantage computation, allowing for adaptive balancing at every stage. Analytical tractability of the advantage function opens possible future directions including entropy-guided regularization or fine-grained difficulty-adaptive optimization schemes. The implicit reward design via advantage provides a framework for dynamic adjustment of learning pressure, suggesting that RLVR algorithms may evolve toward more context-sensitive approaches for reasoning-oriented model development.

7. Conclusion

Pass@k (Pass@$o$) metric advances RLVR by enabling explicit integration of exploration into both evaluation and training regimes for large reasoning models. Use of analytically derived advantage functions eliminates sampling-induced variance and adds efficiency, while the group-based reward design allows exploration and exploitation to mutually reinforce each other. These innovations support the emergence of more adaptive, generalizable, and reasoning-capable LLMs, marking an important step in the evolution of reward design frameworks for large-scale neural reasoning systems.