---
title: 'PASS@(k,T): Stochastic Success Metrics'
url: https://www.emergentmind.com/topics/pass-k-t
type: topic
---

# PASS@(k,T): Stochastic Success Metrics

Searching arXiv for recent papers on PASS@(k,T), Pass@k optimization, and related evaluation frameworks.
PASS@(k,T) denotes a family of pass-based objectives and evaluation metrics built around the probability that a stochastic model produces at least one successful outcome under a constrained sampling or interaction budget. In the recent literature, the notation is not fully standardized: some works use PASS@(k,T) for group-based reinforcement learning over $T$ rollouts with a size-$k$ success criterion or subset reward [2508.10751], some use it for policy-gradient objectives over $T$ sampled trajectories [2510.23049], and some use it as a two-dimensional evaluation surface that jointly varies sampling budget $k$ and interaction depth $T$ for tool-using agents [2604.14877]. Closely related formulations also appear under the more conventional name pass@k, including direct pass@k policy optimization [2505.15201], ranking objectives for code generation [2408.05715], and inference-time selection strategies [2510.03199]. Across these usages, the central theme is the same: pass-based criteria evaluate or optimize the utility of *sets* of attempts rather than isolated single samples.

## 1. Terminological scope and competing definitions

The most common underlying notion is the probability that at least one out of several attempts succeeds. In "Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems" [2505.15201], the binary definition is
$$
\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]
$$
or equivalently
$$
\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].
$$
For $T\ge k$ samples, the paper writes
$$
\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].
$$

"Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models" [2508.10751] uses the notation PASS@(k,T) for the expected maximum reward over $k$ draws, while treating $T$ as the total number of rollouts available for a prompt. In its binary setting,
$$
\mathrm{PASS@(k,T)} = P(\text{at least one of the } k \text{ samples is correct}),
$$
and, conditional on observing $N_{\text{pos}}$ correct samples among $T$ total rollouts, it gives the closed form
$$
\mathrm{PASS@(k,T}\mid N_{\text{pos}})=1-\frac{\binom{N_{\text{neg}}}{k}}{\binom{T}{k}}.
$$

By contrast, "Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis" [2604.14877] defines PASS@(k,T) as a genuinely two-dimensional metric for agentic tool use. There, $k$ is the number of independently sampled trajectories and $T$ is the maximum number of rounds of environment interaction. For a problem $q$ and policy $\pi$,
$$
\mathrm{PASS@(k,T)}(q,\pi)=\Pr[\exists i\in[k]: \tau_i \text{ is correct}],
$$
with an unbiased hypergeometric estimator from $n$ rollouts:
$$
\mathrm{PASS@(k,T)}(q,\pi)=1-\frac{\binom{n-c_T}{k}}{\binom{n}{k}}.
$$

A separate and materially different notation appears in "Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients" [2510.23049], where Pass@(k,T) is defined as the indicator that *at least $k$ of the $T$ samples are correct*:
$$
r_{P@k,T}(\{y_i\}_{i=1}^T,a)=\mathbf{1}\left\{\sum_{i=1}^T r_{0/1}(y_i,a)\ge k\right\}.
$$
This is not the same object as the usual “at least one success among $k$ attempts” metric.

These usages show that PASS@(k,T) is a non-unified notation. A careful reading of any given paper is therefore necessary to determine whether $(k,T)$ refers to subset size versus total rollouts, success count versus sample budget, or sample budget versus interaction depth. This suggests that the notation functions more as a research-local shorthand than as a universally fixed metric name.

## 2. Core probabilistic and combinatorial structure

In the binary case, pass-based metrics reduce to the probability that a sampled set contains at least one successful trajectory. "Learning with a Single Rollout via Monte Carlo Pass@k Critic" [2606.25451] defines the prefix-conditioned single-continuation success probability as
$$
q_\pi(s_t)\coloneqq P_\pi(Y=1\mid s_t),
$$
and the corresponding pass@k probability as
$$
q_{\pi,k}(s)=P\!\left(\max_{1\le i\le k}Y_i=1\mid s\right)=1-(1-q_\pi(s))^k.
$$
When $k=1$, this reduces to Pass@1.

The same “at least one success” quantity can be expressed combinatorially after drawing a larger batch. In [2505.15201], if $n$ samples contain exactly $c$ successes, then
$$
\rho(n,c,k)\coloneqq 1-\frac{\binom{n-c}{k}}{\binom{n}{k}}
$$
is an unbiased estimator of PASS@(k,n). The counting argument is that among the $\binom{n}{k}$ subsets of size $k$, exactly $\binom{n-c}{k}$ contain zero successes.

The same closed form also appears in code-generation ranking. "Top Pass: Improve Code Generation by Pass@k-Maximized Code Ranking" [2408.05715] states that for a random unordered candidate list with $n$ total programs and $c$ correct ones,
$$
\mathbb{E}[\mathrm{pass@}k]=1-\frac{\binom{n-c}{k}}{\binom{n}{k}}.
$$
This is the expected pass@k under random subset selection, and ranking methods aim to improve upon this by moving likely-correct programs into the top-$k$ portion of the list.

Some works also generalize from binary rewards to real-valued rewards. In [2505.15201], the continuous analogue is
$$
\mathrm{PASS@}_k=\mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[\max_{i=1\dots k} g(x_i)\right],
$$
and for $T\ge k$,
$$
\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1\dots x_T}\mathbb{E}_{|I|=k}\!\left[\max_{i\in I} g(x_i)\right].
$$
In "Best-of-Majority: Minimax-Optimal Strategy for Pass@$k$ Inference Scaling" [2510.03199], the inference-time definition is likewise
$$
\mathrm{Pass@}k \equiv \mathbb{E}_{y_1,\dots,y_k\sim\pi}\!\left[\max_{1\le i\le k} r^*(x,y_i)\right],
$$
which coincides with the binary probability-of-success interpretation when $r^*\in\{0,1\}$.

## 3. Policy optimization and analytic estimators

A major line of work treats pass-based quantities not merely as evaluation metrics but as direct reinforcement-learning objectives. The central criticism of conventional RL in [2505.15201] is that sampling multiple $n>1$ attempts and rewarding them independently “optimizes for pass@1 performance” and prioritizes isolated sample strength over diversity and collective utility. The proposed remedy is Pass-at-k Policy Optimization (PKPO), which derives low variance unbiased estimators for pass@k and its gradient in both binary and continuous settings [2505.15201].

For binary rewards, [2505.15201] gives the gradient estimator
$$
\widehat{\nabla}_\theta\,\mathrm{PASS@(k,n)}
=\sum_{i=1}^n r_i \nabla_\theta \log p_\theta(x_i),
$$
with transformed per-sample rewards
$$
r_i=
\begin{cases}
\frac{k}{n}, & \text{if } f_i=1,\\[4pt]
\frac{k}{n}\,\rho(n-1,c,k-1), & \text{if } f_i=0.
\end{cases}
$$
The paper further derives continuous-reward counterparts and two unbiased baselined variants designed for variance reduction. It states that optimization with these estimators reduces to standard RL with rewards jointly transformed by a stable and efficient transformation function, and that the method is the first to enable robust optimization of pass@k for arbitrary $k\le n$ rather than only $k=n$ [2505.15201].

"Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models" [2508.10751] approaches the same general objective from the perspective of analytic advantage design. It computes group-level mean reward
$$
\bar R^{\text{group}}=1-\frac{\binom{N_{\text{neg}}}{k}}{\binom{T}{k}},
$$
and group-level standard deviation
$$
\sigma^{\text{group}}=\sqrt{\bar R^{\text{group}}(1-\bar R^{\text{group}})}.
$$
From this it derives closed-form response-level advantages. For positive samples,
$$
A_{\text{pos}}=\frac{1-\bar R^{\text{group}}}{\sigma^{\text{group}}},
$$
and for negative samples,
$$
A_{\text{neg}}=
\frac{1-\bar R^{\text{group}}-\left(\binom{N_{\text{neg}}-1}{k-1}/\binom{T-1}{k-1}\right)}
{\sigma^{\text{group}}}.
$$
The paper describes this as an efficient and effective process and argues that exploration and exploitation are not inherently conflicting objectives [2508.10751].

A distinct unification is provided by [2510.23049], which argues that direct REINFORCE-style Pass@K optimization and advantage shaping are “two sides of the same coin.” In its special case $k=1$, $T=K$,
$$
\rho_{1,K}(\theta;x,a)=1-(1-\rho_\theta(x,a))^K,
$$
and the paper derives
$$
\nabla_\theta \rho_{1,K}(\theta;x,a)
=
K(1-\rho_\theta)^{K-1}\nabla_\theta \rho_\theta.
$$
It then shows how shaped GRPO-style updates can be interpreted as maximizing surrogate rewards rather than being an unrelated heuristic [2510.23049].

This body of work collectively reframes pass-based RL as optimization over *set-level utility*. A plausible implication is that the main algorithmic challenge is not defining the objective, but constructing low-variance estimators and credit-assignment schemes that preserve unbiasedness or controlled bias while remaining computationally tractable.

## 4. Critic-based credit assignment and the reachability limit

Pass-based objectives have also been adapted to token-level credit assignment without repeated per-prefix sampling. In [2606.25451], single-rollout proximal policy optimization (SR-PPO) trains a calibrated prefix-value critic from one rollout per prompt. The core quantity is a Pass@k critic over prefixes:
$$
q_{\pi,k}(s)=1-(1-q_\pi(s))^k.
$$
The paper trains a prefix-value network with
$$
L_{\text{critic}}(\theta)=\frac{1}{T+1}\sum_{t=0}^T
\left[\mathrm{BCE}(v_\theta(s_t),Y)+(v_\theta(s_t)-Y)^2\right],
$$
where the squared term acts as a Brier-score regularizer [2606.25451].

To target Pass@k directly, [2606.25451] trains a second network and “un-transforms” it via
$$
\hat q_\theta(s)\coloneqq 1-[1-v_{\theta,k}(s)]^{1/k}.
$$
The derivative of the Pass@k map is
$$
\frac{\partial q_{\pi,k}}{\partial q_\pi}=k(1-q_\pi)^{k-1},
$$
which the paper states vanishes as $q_\pi\to 1$ for $k>1$, down-weighting already-easy prefixes [2606.25451]. Its token-level advantage is
$$
A_t^{(k)}=[v_{t,k}-v_{t-1,k}] + \lambda[Y-v_{T,k}],
\qquad \lambda=1.
$$
This is then inserted into the ordinary policy-gradient surrogate
$$
J(\theta)=\mathbb{E}_{s_t,a_t\sim\pi_\theta}
[\log \pi_\theta(a_t\mid s_t)\cdot A_t^{(k)}].
$$

A notable theoretical feature of [2606.25451] is the limit
$$
\lim_{k\to\infty} q_{\pi,k}(s)=r_\pi(s),
$$
where
$$
r_\pi(s)\coloneqq 1\{q_\pi(s)>0\}
$$
is a Boolean reachability indicator. In an explicit state graph, the paper states that this limit can be computed in $O(|V|+|E|)$ time by reverse BFS once the support graph is known. This establishes a connection between large-$k$ pass-based credit assignment and the existence of *any* successful continuation, rather than its single-rollout probability [2606.25451].

This suggests that pass-based critics can interpolate between probability estimation at small $k$ and reachability-style credit assignment at large $k$. That interpretation is explicit in [2606.25451], where larger $k$ is described as yielding a more selective learning signal by discounting easily solved prefixes while prioritizing hard ones whose success probability remains marginal.

## 5. Evaluation, ranking, and inference-time selection

Pass-based quantities also function as operational evaluation criteria outside policy optimization. In code ranking, [2408.05715] treats pass@k as the direct measure of user convenience: a user can test only $k$ programs, so the goal is to maximize the chance that the top-$k$ list contains at least one correct program. It gives the ranking-based condition
$$
\mathrm{pass@}k
=
\mathbf{1}\bigl[f(Q,C^+_{(1)})>f(Q,C^-_{(k)})\bigr],
$$
where $C^+_{(1)}$ is the highest-scored correct sample and $C^-_{(k)}$ the $k$-th highest-scored incorrect sample. Top Pass then optimizes a surrogate pairwise loss over selected positive and negative subsets and reports improvements on four benchmarks, including a 32.9\% relative improvement in pass@1 on CodeContests compared to the state-of-the-art ranking method [2408.05715].

Inference-time selection under a fixed sampling budget is formalized differently in [2510.03199]. There, one first generates $N$ samples from a policy $\pi(\cdot\mid x)$ and then selects a subset of size $k$ for final submission. The paper studies regret
$$
\mathrm{Regret}(x)
=
\mathbb{E}_{y\sim\pi^*}[r^*(x,y)]
-
\mathbb{E}_{y_1,\dots,y_k}\!\left[\max_{1\le i\le k} r^*(x,y_i)\right],
$$
and, for a unique maximizer, states that
$$
\mathrm{Pass@}k = 1-\mathrm{Regret}(x).
$$
Its Best-of-Majority (BoM) algorithm filters candidates by empirical frequency and then ranks survivors by reward-model score. The paper proves that neither majority voting nor Best-of-$N$ exhibits the desired scaling with $k$ and $N$, while BoM is minimax-optimal up to constant and log factors [2510.03199].

A more critical stance appears in "Don't Pass$\mathtt{@}k$: A Bayesian Framework for Large Language Model Evaluation" [2510.04265]. That work argues that Pass$@k$ often yields unstable, misleading rankings when the number of trials is limited. It replaces point estimates with posterior estimates of the underlying success probability and credible intervals, using a Dirichlet prior over categorical outcomes. In the binary case with a uniform prior, the posterior mean of the single-draw success probability is
$$
\mu_1=\frac{1+n_1}{2+k},
$$
and the paper defines a Bayesian PASS@(k,T) as the posterior expectation of
$$
1-(1-\pi_\alpha)^k
$$
averaged over problems [2510.04265]. It further notes that, under a uniform prior, the Bayesian posterior mean is order-equivalent to average accuracy (Pass@1), which it presents as an explanation for the empirical robustness of average accuracy ranking [2510.04265].

Together these works show that pass-based metrics serve at least three distinct evaluative roles: ranking top-$k$ outputs for user inspection, selecting a size-$k$ inference set from a larger candidate pool, and summarizing a model’s latent success probability under uncertainty. A plausible implication is that “pass@k” should be interpreted as an operational metric whose meaning depends strongly on the surrounding decision problem.

## 6. Interaction depth, diversity, and capability boundaries

The most explicit use of PASS@(k,T) as a two-dimensional object appears in [2604.14877], where $k$ controls resampling and $T$ controls interaction depth. This metric is introduced to distinguish “efficiency improvements” from “capability expansion” in tool-using agents. The paper states two useful limits: as $k\to n$, PASS@(k,T) approaches the indicator that the problem lies inside the capability boundary at depth $T$, and when $T=0$ it reduces exactly to standard static pass@k [2604.14877].

Empirically, [2604.14877] reports that for static reasoning, base and RL pass@k curves converge at large $k$, whereas for compositional tool use the RL agent’s pass-curve “pulls above the base model’s and the gap widens at large $k$ rather than converging.” On HotPotQA bridge questions, the paper reports at $k=64$, $T=5$:
- $|\mathcal{B}_5(\pi_{\text{base}})|=77$
- $|\mathcal{B}_5(\pi_{\text{SFT}})|=73$
- $|\mathcal{B}_5(\pi_{\text{RL}})|=81$

It further reports that on the same bridge questions, $\pi_{\text{RL}}$ solves 5 problems the base cannot, while $\pi_{\text{SFT}}$ loses ground relative to the base despite identical training data [2604.14877]. The paper interprets this as evidence that self-directed exploration, rather than data exposure alone, is the causal factor.

Related work on diversity also connects large-$k$ gains to broader coverage of reasoning modes. "Transform-Augmented GRPO Improves Pass@k" [2601.22478] defines
$$
\mathrm{Pass@}k(q)=1-(1-\rho_q)^k
$$
for a per-sample success rate $\rho_q$, and proves that transform-augmentation reduces zero-gradient probability and can strictly increase the pooled success rate when any transformed variant is easier than the original. Its experiments report gains up to 9.84 points on AMC12, 8.69 on AIME24, and 5.05 on GPQA-Diamond in Pass@32 at temperature $0.7$ [2601.22478].

A more explicitly coordinated strategy appears in "Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning" [2605.27000], which argues that i.i.d. sampling from a single answer distribution can collapse onto near-duplicate reasoning paths. CPPO replaces $K$ independent draws with a planner that emits a tuple of $K=4$ alternative high-level methods and a shared solver that attempts one solution per method. With the multiplicative planner reward
$$
R_{\mathrm{plan}}=J_\psi\cdot R_{\mathrm{out}},
$$
the paper reports statistically significant pass@4 gains on six of nine model-benchmark cells, including a largest single gain of $+0.16$ on Qwen3.5-9B LiveCodeBench-v6 over PKPO, from $0.588$ to $0.748$ [2605.27000].

These results reinforce a recurring theme: large-$k$ performance depends not only on per-sample correctness but also on the structure of diversity across attempts. This suggests that PASS@(k,T) is especially informative when additional samples or interaction rounds can expose qualitatively different solution strategies rather than merely repeated draws from a narrow mode.

## 7. Conceptual significance and unresolved standardization

Across the cited literature, pass-based objectives serve as a bridge between evaluation, inference, and reinforcement learning. They shift focus from isolated trajectories to the utility of *sets* of trajectories, and they make explicit the role of computational budget—whether measured as sample count, total rollouts, or interaction depth—in determining success [2505.15201].

At the same time, the notation PASS@(k,T) remains unsettled. The following usages all appear in recent papers:

| Usage | Meaning of $k$ | Meaning of $T$ |
|---|---:|---:|
| [2508.10751] | group size / tested subset | total rollouts |
| [2604.14877] | number of sampled trajectories | interaction depth |
| [2510.23049] | minimum number of correct samples | total sampled responses |

This lack of uniformity is itself an important fact about the topic. It means that PASS@(k,T) is not yet a single canonical metric but a notation family built around pass-based success under resource constraints.

A second unresolved issue concerns whether pass-based evaluation is the right instrument for model comparison. [2510.04265] argues that Pass@k can be unstable and misleading under limited trials and advocates posterior estimates of latent success probability with credible intervals instead. By contrast, optimization papers such as [2505.15201] and [2508.10751] treat pass-based objectives as central training signals precisely because they better align with multi-sample deployment and exploration.

A third issue concerns what large-$k$ gains should mean scientifically. In static reasoning, convergence of base and RL pass-curves at high $k$ has been interpreted as evidence that RL mainly improves reliability; in agentic tool use, [2604.14877] reports the opposite pattern and interprets it as capability expansion. This suggests that the semantics of pass-based improvements depend on whether additional samples merely reweight existing strategies or unlock new compositional behaviors.

PASS@(k,T), in its various forms, therefore occupies a distinctive role in current language-model research: it is simultaneously a metric of best-of-many success, a family of RL objectives for set-level optimization, and a diagnostic for the interaction between exploration, diversity, and bounded compute. Its continuing development is likely to depend less on the basic probability formula than on clarifying notation, estimator design, and the scientific interpretation of gains along the $k$ and $T$ axes.

Source: https://www.emergentmind.com/topics/pass-k-t