PASS@(k,T): Stochastic Success Metrics

• PASS@(k,T) is a pass-based metric family that quantifies the probability of achieving at least one success across multiple attempts under constrained sampling budgets.
• It captures key aspects such as sampling budget, interaction depth, and varied definitions across contexts like reinforcement learning and code generation.
• It informs both evaluation and optimization by guiding diversity in sample selection and enabling low-variance credit assignment in multi-sample environments.

Searching arXiv for papers on PASS@(k,T), Pass@k optimization, and related evaluation frameworks. PASS@(k,T) denotes a family of pass-based objectives and evaluation metrics built around the probability that a stochastic model produces at least one successful outcome under a constrained sampling or interaction budget. In the recent literature, the notation is not fully standardized: some works use PASS@(k,T) for group-based reinforcement learning over $T$ rollouts with a size-$k$ success criterion or subset reward (Chen et al., 14 Aug 2025), some use it for policy-gradient objectives over $T$ sampled trajectories (Thrampoulidis et al., 27 Oct 2025), and some use it as a two-dimensional evaluation surface that jointly varies sampling budget $k$ and interaction depth $T$ for tool-using agents (Zhai et al., 16 Apr 2026). Closely related formulations also appear under the more conventional name pass@k, including direct pass@k policy optimization (Walder et al., 21 May 2025), ranking objectives for code generation (Lyu et al., 2024), and inference-time selection strategies (Di et al., 3 Oct 2025). Across these usages, the central theme is the same: pass-based criteria evaluate or optimize the utility of sets of attempts rather than isolated single samples.

1. Terminological scope and competing definitions

The most common underlying notion is the probability that at least one out of several attempts succeeds. In "Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems" (Walder et al., 21 May 2025), the binary definition is

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$

or equivalently

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$

For $T\ge k$ samples, the paper writes

$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$

"Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models" (Chen et al., 14 Aug 2025) uses the notation PASS@(k,T) for the expected maximum reward over $k$ draws, while treating $k$0 as the total number of rollouts available for a prompt. In its binary setting,

$k$1

and, conditional on observing $k$2 correct samples among $k$3 total rollouts, it gives the closed form

$k$4

By contrast, "Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis" (Zhai et al., 16 Apr 2026) defines PASS@(k,T) as a genuinely two-dimensional metric for agentic tool use. There, $k$5 is the number of independently sampled trajectories and $k$6 is the maximum number of rounds of environment interaction. For a problem $k$7 and policy $k$8,

$k$9

with an unbiased hypergeometric estimator from $T$0 rollouts:

$T$1

A separate and materially different notation appears in "Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients" (Thrampoulidis et al., 27 Oct 2025), where Pass@(k,T) is defined as the indicator that at least $T$2 of the $T$3 samples are correct:

$T$4

This is not the same object as the usual “at least one success among $T$5 attempts” metric.

These usages show that PASS@(k,T) is a non-unified notation. A careful reading of any given paper is therefore necessary to determine whether $T$6 refers to subset size versus total rollouts, success count versus sample budget, or sample budget versus interaction depth. This suggests that the notation functions more as a research-local shorthand than as a universally fixed metric name.

2. Core probabilistic and combinatorial structure

In the binary case, pass-based metrics reduce to the probability that a sampled set contains at least one successful trajectory. "Learning with a Single Rollout via Monte Carlo Pass@k Critic" (Che et al., 24 Jun 2026) defines the prefix-conditioned single-continuation success probability as

$T$7

and the corresponding pass@k probability as

$T$8

When $T$9, this reduces to Pass@1.

The same “at least one success” quantity can be expressed combinatorially after drawing a larger batch. In (Walder et al., 21 May 2025), if $k$0 samples contain exactly $k$1 successes, then

$k$2

is an unbiased estimator of PASS@(k,n). The counting argument is that among the $k$3 subsets of size $k$4, exactly $k$5 contain zero successes.

The same closed form also appears in code-generation ranking. "Top Pass: Improve Code Generation by Pass@k-Maximized Code Ranking" (Lyu et al., 2024) states that for a random unordered candidate list with $k$6 total programs and $k$7 correct ones,

$k$8

This is the expected pass@k under random subset selection, and ranking methods aim to improve upon this by moving likely-correct programs into the top-$k$9 portion of the list.

Some works also generalize from binary rewards to real-valued rewards. In (Walder et al., 21 May 2025), the continuous analogue is

$T$0

and for $T$1,

$T$2

In "Best-of-Majority: Minimax-Optimal Strategy for Pass@$T$3 Inference Scaling" (Di et al., 3 Oct 2025), the inference-time definition is likewise

$T$4

which coincides with the binary probability-of-success interpretation when $T$5.

3. Policy optimization and analytic estimators

A major line of work treats pass-based quantities not merely as evaluation metrics but as direct reinforcement-learning objectives. The central criticism of conventional RL in (Walder et al., 21 May 2025) is that sampling multiple $T$6 attempts and rewarding them independently “optimizes for pass@1 performance” and prioritizes isolated sample strength over diversity and collective utility. The proposed remedy is Pass-at-k Policy Optimization (PKPO), which derives low variance unbiased estimators for pass@k and its gradient in both binary and continuous settings (Walder et al., 21 May 2025).

For binary rewards, (Walder et al., 21 May 2025) gives the gradient estimator

$T$7

with transformed per-sample rewards

$T$8

The paper further derives continuous-reward counterparts and two unbiased baselined variants designed for variance reduction. It states that optimization with these estimators reduces to standard RL with rewards jointly transformed by a stable and efficient transformation function, and that the method is the first to enable robust optimization of pass@k for arbitrary $T$9 rather than only $$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$0 (Walder et al., 21 May 2025).

"Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models" (Chen et al., 14 Aug 2025) approaches the same general objective from the perspective of analytic advantage design. It computes group-level mean reward

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$1

and group-level standard deviation

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$2

From this it derives closed-form response-level advantages. For positive samples,

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$3

and for negative samples,

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$4

The paper describes this as an efficient and effective process and argues that exploration and exploitation are not inherently conflicting objectives (Chen et al., 14 Aug 2025).

A distinct unification is provided by (Thrampoulidis et al., 27 Oct 2025), which argues that direct REINFORCE-style Pass@K optimization and advantage shaping are “two sides of the same coin.” In its special case $$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$5, $$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$6,

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$7

and the paper derives

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$8

It then shows how shaped GRPO-style updates can be interpreted as maximizing surrogate rewards rather than being an unrelated heuristic (Thrampoulidis et al., 27 Oct 2025).

This body of work collectively reframes pass-based RL as optimization over set-level utility. A plausible implication is that the main algorithmic challenge is not defining the objective, but constructing low-variance estimators and credit-assignment schemes that preserve unbiasedness or controlled bias while remaining computationally tractable.

4. Critic-based credit assignment and the reachability limit

Pass-based objectives have also been adapted to token-level credit assignment without repeated per-prefix sampling. In (Che et al., 24 Jun 2026), single-rollout proximal policy optimization (SR-PPO) trains a calibrated prefix-value critic from one rollout per prompt. The core quantity is a Pass@k critic over prefixes:

$$\mathrm{PASS@}_k = P\!\left[\bigvee_{i=1}^k (f(x_i)=1)\right]$$9

The paper trains a prefix-value network with

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$0

where the squared term acts as a Brier-score regularizer (Che et al., 24 Jun 2026).

To target Pass@k directly, (Che et al., 24 Jun 2026) trains a second network and “un-transforms” it via

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$1

The derivative of the Pass@k map is

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$2

which the paper states vanishes as $$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$3 for $$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$4, down-weighting already-easy prefixes (Che et al., 24 Jun 2026). Its token-level advantage is

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$5

This is then inserted into the ordinary policy-gradient surrogate

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$6

A notable theoretical feature of (Che et al., 24 Jun 2026) is the limit

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$7

where

$$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$8

is a Boolean reachability indicator. In an explicit state graph, the paper states that this limit can be computed in $$\mathrm{PASS@}_k = \mathbb{E}_{x_1,\dots,x_k\sim p_\theta}\!\left[1-\prod_{i=1}^k (1-f(x_i))\right].$$9 time by reverse BFS once the support graph is known. This establishes a connection between large-$T\ge k$0 pass-based credit assignment and the existence of any successful continuation, rather than its single-rollout probability (Che et al., 24 Jun 2026).

This suggests that pass-based critics can interpolate between probability estimation at small $T\ge k$1 and reachability-style credit assignment at large $T\ge k$2. That interpretation is explicit in (Che et al., 24 Jun 2026), where larger $T\ge k$3 is described as yielding a more selective learning signal by discounting easily solved prefixes while prioritizing hard ones whose success probability remains marginal.

5. Evaluation, ranking, and inference-time selection

Pass-based quantities also function as operational evaluation criteria outside policy optimization. In code ranking, (Lyu et al., 2024) treats pass@k as the direct measure of user convenience: a user can test only $T\ge k$4 programs, so the goal is to maximize the chance that the top-$T\ge k$5 list contains at least one correct program. It gives the ranking-based condition

$T\ge k$6

where $T\ge k$7 is the highest-scored correct sample and $T\ge k$8 the $T\ge k$9-th highest-scored incorrect sample. Top Pass then optimizes a surrogate pairwise loss over selected positive and negative subsets and reports improvements on four benchmarks, including a 32.9\% relative improvement in pass@1 on CodeContests compared to the state-of-the-art ranking method (Lyu et al., 2024).

Inference-time selection under a fixed sampling budget is formalized differently in (Di et al., 3 Oct 2025). There, one first generates $$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$0 samples from a policy $$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$1 and then selects a subset of size $$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$2 for final submission. The paper studies regret

$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$3

and, for a unique maximizer, states that

$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$4

Its Best-of-Majority (BoM) algorithm filters candidates by empirical frequency and then ranks survivors by reward-model score. The paper proves that neither majority voting nor Best-of-$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$5 exhibits the desired scaling with $$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$6 and $$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$7, while BoM is minimax-optimal up to constant and log factors (Di et al., 3 Oct 2025).

A more critical stance appears in "Don't Pass$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$8: A Bayesian Framework for LLM Evaluation" (Hariri et al., 5 Oct 2025). That work argues that Pass$$\mathrm{PASS@(k,T)}=\mathbb{E}_{x_1,\dots,x_T}\mathbb{E}_{|I|=k}\!\left[1-\prod_{i\in I}(1-f(x_i))\right].$$9 often yields unstable, misleading rankings when the number of trials is limited. It replaces point estimates with posterior estimates of the underlying success probability and credible intervals, using a Dirichlet prior over categorical outcomes. In the binary case with a uniform prior, the posterior mean of the single-draw success probability is

$k$0

and the paper defines a Bayesian PASS@(k,T) as the posterior expectation of

$k$1

averaged over problems (Hariri et al., 5 Oct 2025). It further notes that, under a uniform prior, the Bayesian posterior mean is order-equivalent to average accuracy (Pass@1), which it presents as an explanation for the empirical robustness of average accuracy ranking (Hariri et al., 5 Oct 2025).

Together these works show that pass-based metrics serve at least three distinct evaluative roles: ranking top-$k$2 outputs for user inspection, selecting a size-$k$3 inference set from a larger candidate pool, and summarizing a model’s latent success probability under uncertainty. A plausible implication is that “pass@k” should be interpreted as an operational metric whose meaning depends strongly on the surrounding decision problem.

6. Interaction depth, diversity, and capability boundaries

The most explicit use of PASS@(k,T) as a two-dimensional object appears in (Zhai et al., 16 Apr 2026), where $k$4 controls resampling and $k$5 controls interaction depth. This metric is introduced to distinguish “efficiency improvements” from “capability expansion” in tool-using agents. The paper states two useful limits: as $k$6, PASS@(k,T) approaches the indicator that the problem lies inside the capability boundary at depth $k$7, and when $k$8 it reduces exactly to standard static pass@k (Zhai et al., 16 Apr 2026).

Empirically, (Zhai et al., 16 Apr 2026) reports that for static reasoning, base and RL pass@k curves converge at large $k$9, whereas for compositional tool use the RL agent’s pass-curve “pulls above the base model’s and the gap widens at large $k$00 rather than converging.” On HotPotQA bridge questions, the paper reports at $k$01, $k$02:

• $k$03
• $k$04
• $k$05

It further reports that on the same bridge questions, $k$06 solves 5 problems the base cannot, while $k$07 loses ground relative to the base despite identical training data (Zhai et al., 16 Apr 2026). The paper interprets this as evidence that self-directed exploration, rather than data exposure alone, is the causal factor.

Related work on diversity also connects large-$k$08 gains to broader coverage of reasoning modes. "Transform-Augmented GRPO Improves Pass@k" (Le et al., 30 Jan 2026) defines

$k$09

for a per-sample success rate $k$10, and proves that transform-augmentation reduces zero-gradient probability and can strictly increase the pooled success rate when any transformed variant is easier than the original. Its experiments report gains up to 9.84 points on AMC12, 8.69 on AIME24, and 5.05 on GPQA-Diamond in Pass@32 at temperature $k$11 (Le et al., 30 Jan 2026).

A more explicitly coordinated strategy appears in "Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning" (Li et al., 26 May 2026), which argues that i.i.d. sampling from a single answer distribution can collapse onto near-duplicate reasoning paths. CPPO replaces $k$12 independent draws with a planner that emits a tuple of $k$13 alternative high-level methods and a shared solver that attempts one solution per method. With the multiplicative planner reward

$k$14

the paper reports statistically significant pass@4 gains on six of nine model-benchmark cells, including a largest single gain of $k$15 on Qwen3.5-9B LiveCodeBench-v6 over PKPO, from $k$16 to $k$17 (Li et al., 26 May 2026).

These results reinforce a recurring theme: large-$k$18 performance depends not only on per-sample correctness but also on the structure of diversity across attempts. This suggests that PASS@(k,T) is especially informative when additional samples or interaction rounds can expose qualitatively different solution strategies rather than merely repeated draws from a narrow mode.

7. Conceptual significance and unresolved standardization

Across the cited literature, pass-based objectives serve as a bridge between evaluation, inference, and reinforcement learning. They shift focus from isolated trajectories to the utility of sets of trajectories, and they make explicit the role of computational budget—whether measured as sample count, total rollouts, or interaction depth—in determining success (Walder et al., 21 May 2025).

At the same time, the notation PASS@(k,T) remains unsettled. The following usages all appear in papers:

Usage	Meaning of $k$19	Meaning of $k$20
(Chen et al., 14 Aug 2025)	group size / tested subset	total rollouts
(Zhai et al., 16 Apr 2026)	number of sampled trajectories	interaction depth
(Thrampoulidis et al., 27 Oct 2025)	minimum number of correct samples	total sampled responses

This lack of uniformity is itself an important fact about the topic. It means that PASS@(k,T) is not yet a single canonical metric but a notation family built around pass-based success under resource constraints.

A second unresolved issue concerns whether pass-based evaluation is the right instrument for model comparison. (Hariri et al., 5 Oct 2025) argues that Pass@k can be unstable and misleading under limited trials and advocates posterior estimates of latent success probability with credible intervals instead. By contrast, optimization papers such as (Walder et al., 21 May 2025) and (Chen et al., 14 Aug 2025) treat pass-based objectives as central training signals precisely because they better align with multi-sample deployment and exploration.

A third issue concerns what large-$k$21 gains should mean scientifically. In static reasoning, convergence of base and RL pass-curves at high $k$22 has been interpreted as evidence that RL mainly improves reliability; in agentic tool use, (Zhai et al., 16 Apr 2026) reports the opposite pattern and interprets it as capability expansion. This suggests that the semantics of pass-based improvements depend on whether additional samples merely reweight existing strategies or unlock new compositional behaviors.

PASS@(k,T), in its various forms, therefore occupies a distinctive role in current language-model research: it is simultaneously a metric of best-of-many success, a family of RL objectives for set-level optimization, and a diagnostic for the interaction between exploration, diversity, and bounded compute. Its continuing development is likely to depend less on the basic probability formula than on clarifying notation, estimator design, and the scientific interpretation of gains along the $k$23 and $k$24 axes.