Cost-of-Pass: A Unified Resource Efficiency Metric

• Cost-of-Pass is a metric that quantifies the resource required to achieve a correct outcome by balancing per-attempt cost with success probability.
• It is applied across various domains—including language modeling, code reasoning, and compiler optimization—to assess the efficiency trade-offs in achieving success.
• The framework benchmarks technological progress and informs practical decisions by comparing model-level innovations with inference-time and computational costs.

“Cost-of-Pass” denotes a family of evaluation concepts centered on the resource required to obtain a successful outcome under an explicit success criterion. In its most formal recent usage, it is the expected monetary cost of generating a correct solution, $v(m,p)=C_m(p)/R_m(p)$, where $C_m(p)$ is expected per-attempt cost and $R_m(p)$ is success probability for model $m$ on problem $p$ (Erol et al., 17 Apr 2025). In verifier-based code generation, the same idea appears as the compute required to obtain at least one verifier-passing program under a fixed attempt budget, with pass@$K$ serving as the operative proxy (Li et al., 26 May 2026). In single-pass retrieval, streaming clustering, randomized PCA, and compiler optimization, the term is used more operationally to denote the computational expense of one full scan or the in-context marginal effect of appending one optimization pass (Cao et al., 4 Apr 2025, Kightley et al., 2019, Yu et al., 2017, Bruzzone et al., 30 Jun 2026). Across these usages, the common structure is a priced attempt or pass, a binary or thresholded notion of success, and an efficiency question about how much resource must be consumed before success occurs.

1. Production-theoretic formulation for LLMs

The most explicit formalization treats LLMs as stochastic producers that transform priced inputs into correct outputs. For a problem $p$ and model or inference pipeline $m$, the paper “Cost-of-Pass: An Economic Framework for Evaluating LLMs” defines $R_m(p)$ as the probability of a correct answer and $C_m(p)$ as expected cost of one inference attempt, then defines cost-of-pass as

$C_m(p)$0

Under repeated independent attempts, this is the expected monetary cost of obtaining one correct solution. The same framework defines the model frontier

$C_m(p)$1

a combined frontier including a human expert baseline,

$C_m(p)$2

and temporal gains from new releases,

$C_m(p)$3

The framework is grounded in production theory: models are stochastic producers, tokens are priced inputs, and the relevant efficiency object is expected dollars per successful output rather than accuracy alone (Erol et al., 17 Apr 2025).

This formulation is benchmark- and task-specific rather than universal. The paper estimates per-sample inference cost from token counts and API prices,

$C_m(p)$4

and then evaluates frontier cost-of-pass over six datasets grouped into basic quantitative tasks, knowledge-based tasks, and complex quantitative reasoning. It also integrates approximate human-expert costs such as \$C_m(p)$50.02 for Two-Digit Addition, \$C_m(p)$69–\$20 for AIME, thereby turning model evaluation into a direct comparison against labor cost rather than a model-only leaderboard.

2. Frontier behavior, model classes, and inference-time tradeoffs

The empirical frontier is heterogeneous across task domains. The paper reports that lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive tasks, and reasoning models for complex quantitative problems, despite higher per-token costs (Erol et al., 17 Apr 2025).

Task category	Representative datasets	Frontier-leading class
Basic quantitative	Two Digit Addition, GSM8K	Lightweight models
Knowledge-based	BBQ, GPQA Diamond	Large models
Complex quantitative reasoning	MATH500, AIME24	Reasoning models

The numerical frontier values illustrate the asymmetry between per-attempt cost and success probability. On Two Digit Addition, frontier values include $C_m(p)$7 for Llama-3.1-8B and $C_m(p)$8 for GPT-4o mini, reflecting the dominance of low-cost models once accuracy is saturated. On BBQ, large models cluster near the frontier, with GPT-4o at $C_m(p)$9, Claude Sonnet-3.5 at $R_m(p)$0, and Llama-3.1-405B at $R_m(p)$1. On MATH500 and AIME24, reasoning models dominate: DeepSeek-R1 reaches $R_m(p)$2 on MATH500, and o3-mini reaches $R_m(p)$3 on AIME24. The paper also fits temporal frontier progress with

$R_m(p)$4

with halving time $R_m(p)$5, and reports that frontier cost-of-pass on MATH500 halved approximately every $R_m(p)$6 months and on AIME24 approximately every $R_m(p)$7 months.

The same framework evaluates inference-time techniques as modified pipelines $R_m(p)$8 with altered $R_m(p)$9 and $m$0. Majority voting with $m$1 or $m$2 produces zero relative frontier improvement across all reported datasets, while Self-Refine yields nonzero gains only on some knowledge-intensive tasks, including $m$3 on GPQA Diamond and $m$4 on BBQ. The paper’s conclusion is that marginal accuracy gains from common inference-time techniques rarely justify their additional cost; frontier movement is driven primarily by model-level innovation rather than by expensive test-time scaffolds.

3. Verifier-budget cost-of-pass in code reasoning and reinforcement learning

In code generation, cost-of-pass is tied to verifier-confirmed success under a finite attempt budget. “Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoning” starts from the standard pass@$m$5 reward

$m$6

so that, at fixed $m$7, higher pass@$m$8 means fewer attempts are needed on average to obtain a pass. The paper argues that the standard iid policy class,

$m$9

wastes budget on duplicate reasoning modes. Its motivating bound is the iid miss probability $p$0; for $p$1, even $p$2 leaves a failure probability of about $p$3. CPPO replaces iid answer draws with a coordinated tuple of strategies

$p$4

and rewards the planner only when a valid, non-degenerate tuple leads to verifier-confirmed pass@$p$5 success: $p$6 In the default $p$7 setting, CPPO improves pass@4 over direct sampling, planning baselines, planner-only SFT, and PKPO under the same solver-attempt budget, with the largest reported gain $p$8 on Qwen3.5-9B LiveCodeBench-v6, $p$9 (Li et al., 26 May 2026).

A closely related RL formulation appears in “Learning with a Single Rollout via Monte Carlo Pass@k Critic,” where repeated sampling is treated as the principal expense in token-level credit assignment. The paper defines prefix Pass@1 value

$K$0

and prefix Pass@$K$1 value

$K$2

Instead of estimating advantages from multiple completions of the same prompt, it trains a calibrated prefix critic from one rollout per prompt and one terminal label. Token-level advantages are constructed from local changes in predicted prefix value plus a terminal correction,

$K$3

The paper reports that each GRPO step consumes $K$4 rollouts, while each SR-PPO step consumes only $K$5, and argues that Pass@4 yields a more selective signal than Pass@1. It also proves that as $K$6, Pass@$K$7 converges to a reachability indicator and that, in an explicit state graph, the limit can be computed in $K$8 time (Che et al., 24 Jun 2026).

4. Single-pass computation as a resource measure

A separate computational usage interprets cost-of-pass as the expense of one complete scan of an input stream or document. In “Single-Pass Document Scanning for Question Answering,” a single pass means concatenating query and document as $K$9, tokenizing once, scanning left-to-right exactly once with a Mamba-2 state-space model, scoring sentences at sentence boundaries, and selecting top-$p$0 sentences, typically $p$1. The paper emphasizes linear scaling in sequence length and reports concrete per-document costs: SPScanner 130M requires $p$2 ms and $p$3 TFLOPs, while SPScanner 1.3B requires $p$4 ms and $p$5 TFLOPs. These costs are lower than several chunk-based embedding baselines and vastly below the reported $p$6 TFLOPs for Llama-3.1-70B direct answer on long documents (Cao et al., 4 Apr 2025).

In streaming clustering, “One-Pass Sparsified Gaussian Mixtures” makes the same idea explicit: each $p$7-dimensional point is preconditioned once, sketch-compressed to $p$8, and then discarded. The paper states an up-front preprocessing cost of $p$9, compressed-data storage of $m$0, and per-iteration EM cost of $m$1 for diagonal or spherical covariances, instead of $m$2. Its one-pass property relies on using a different subsampling matrix $m$3 for each point, allowing estimation of full $m$4-dimensional parameters without revisiting the raw data (Kightley et al., 2019).

The same pass-efficiency logic appears in large-scale PCA. “Single-Pass PCA of Large High-Dimensional Data” defines a pass as a complete scan of the data matrix $m$5, often from disk. The proposed algorithm accumulates

$m$6

during a single scan, then finishes all subsequent factorization work in memory. The paper reports complexity $m$7 or $m$8, memory $m$9, and an out-of-core demonstration computing the first $R_m(p)$0 principal components of a $R_m(p)$1 GB file in $R_m(p)$2 minutes with less than $R_m(p)$3 GB memory (Yu et al., 2017).

5. Per-pass marginal cost in compiler optimization

In compiler optimization, cost-of-pass denotes the in-context marginal effect of appending one pass to a cumulative optimization prefix. “A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline” flattens LLVM -O3 into pass invocations $R_m(p)$4, constructs cumulative prefixes

$R_m(p)$5

and defines the marginal runtime effect of adding pass $R_m(p)$6 as

$R_m(p)$7

The study evaluates $R_m(p)$8 cumulative prefixes over $R_m(p)$9 PolyBench/C kernels and reports $C_m(p)$0 measurements. Its central result is that pass cost is intrinsically contextual: the same pass occurrence can improve runtime, regress runtime, change compile time, inflate binary size, alter hardware counters, or reduce energy, depending on its exact location in the canonical pipeline (Bruzzone et al., 30 Jun 2026).

The empirical picture is strongly non-monotone and back-loaded. Using the mean estimator, $C_m(p)$1 of informative pass transitions regress runtime; using the median estimator, $C_m(p)$2 regress runtime. Among non-regressing kernels, the median kernel requires $C_m(p)$3 of the pipeline to reach $C_m(p)$4 of its final speedup. Most gains are concentrated in a small Pareto-dominant core of pass occurrences, led by early-cse, licm, instcombine, loop-vectorize, loop-rotate, loop-unroll, simplifycfg, sroa, and indvars. The final -O3 endpoint is Pareto-dominated on $C_m(p)$5 for $C_m(p)$6 of $C_m(p)$7 kernels. The paper also reports that package energy falls from $C_m(p)$8 J to $C_m(p)$9 J, while IPC falls from $C_m(p)$00 to $C_m(p)$01, implying that runtime-targeted passes are de facto energy-targeted on these compute-bound kernels even when instruction throughput declines.

6. Related but distinct meanings: cost pass-through and passive implementation

A distinct adjacent literature uses “pass” in the sense of pass-through rather than expected cost to success. In this usage, the question is how much of an exogenous cost shock is transmitted into prices, wages, amenities, or other outcomes. In a Japanese spot labor platform, minimum wage hikes produced a $C_m(p)$02 percentage point decline in employment within affected wage bins, while transportation reimbursement remained largely unchanged, which the paper interprets as limited amenity pass-through and cost burden borne by establishments (Kanayama et al., 7 May 2025). In Washington cannabis, wholesale minimum-wage pass-through elasticity is reported as $C_m(p)$03, retail direct elasticity as $C_m(p)$04, retail indirect elasticity as $C_m(p)$05, and retail total elasticity as $C_m(p)$06, showing that upstream wholesale costs more than double the retail pass-through estimated from retail data alone (Hase, 2023).

Carbon and electricity-market studies use the same language. For Italy under the EU ETS, national carbon cost pass-through remains relatively stable at around $C_m(p)$07 across Phases 3 and 4, with large zonal heterogeneity: pass-through rises in North, Centre-North, and Sardinia, and declines in Centre-South and Sicily (Duttilo et al., 3 Apr 2026). In an Austrian-German 2017 power-system model, the estimated pass-through from CO$C_m(p)$08 prices to power prices lies between $C_m(p)$09 and $C_m(p)$10, and the paper argues that low-emission district heating systems with sufficiently efficient CHP units can benefit from higher CO$C_m(p)$11 prices in the short run (Wehrle et al., 2018). Theoretical work then generalizes pass-through itself: “Pass-through with Price Dispersion” decomposes it into a competition layer and a curvature layer through normalized effective margins, while “Demand Curvature and Pass-Through in Differentiated Oligopoly” derives the pass-through matrix

$C_m(p)$12

thereby decomposing price responses into demand curvature, substitution, and multiproduct ownership (Albrecht et al., 25 Jan 2026, Koh, 23 Apr 2026). “Multi-Dimensional Pass-Through and Welfare Measures under Imperfect Competition” further links multi-instrument pass-through to marginal cost of public funds and incidence (Adachi et al., 2017).

Finance supplies another adjacent meaning in which the cost of “pass” is the cost of mechanically adhering to passive implementation rules. “On the hidden costs of passive investing” argues that waiting until market close on reconstitution day to buy an announced index addition can cost hundreds of basis points relative to gradual pre-positioning with minimal additional tracking error. A plausible implication is that several recent literatures use closely related language to study the economic burden of mechanical rules, even when the object of interest is no longer a verifier pass or a single computational scan (Tasitsiomi, 26 Jun 2025).

7. Conceptual unification and scope conditions

Across these literatures, cost-of-pass is not a single invariant metric but a reusable abstraction. One version measures expected dollars per correct answer, another measures verifier-budget required for at least one passing program, another measures the computational expense of one full scan, and another measures the in-context marginal contribution of an optimization pass. What unifies them is the structure of a costly act, a pass criterion, and a frontier or marginal trade-off between expenditure and success.

The differences are equally important. In the language-model formulation, cost-of-pass is formally defined and directly monetized. In code reasoning, it is usually inferred from pass@$C_m(p)$13 at fixed $C_m(p)$14 rather than measured as a separate expected-attempts statistic. In single-pass algorithms, the “pass” is a physical scan of data, and the relevant quantities are latency, FLOPs, memory, or I/O complexity. In compiler studies, the “pass” is a transformation stage whose cost is only meaningful relative to pipeline context. This suggests that the term is best treated as a family resemblance concept rather than a single equation: it denotes the resource price of reaching a desired event under constraints imposed by stochasticity, verification, or sequential processing.