Scaled Pass@1 Score in LLM Evaluation

• Scaled Pass@1 is a Bayesian-based metric that estimates the true success rate of LLM outputs using a conjugate Beta prior to mitigate extreme values in low-sample settings.
• It combines observed successes with a uniform prior to generate credible intervals, offering robust model comparisons and avoiding pathological cases seen with raw Pass@1.
• The metric generalizes to Pass@k evaluations, making it widely applicable for tasks like reasoning and code generation by providing interpretable estimates and quantifiable uncertainty.

A scaled Pass@1 score is a statistically principled metric for evaluating the single-sample success probability of LLMs on binary verifiable tasks. Unlike raw Pass@1—which is simply the observed proportion of correct responses in $N$ trials—scaled Pass@1 adopts a Bayesian approach, combining observed data with a conjugate Beta prior to provide robust point estimates, controlled shrinkage in low-sample regimes, and credible intervals for model comparison. The metric generalizes naturally to other Pass@$k$-type metrics, provides a unified framework for ranking and reporting, and addresses several known pitfalls of naïve or plug-in estimators when evaluating LLM performance.

1. Formal Definition and Bayesian Framework

Let $N$ denote the number of independent evaluation trials, and $C$ the number of successful outcomes ($R_i = 1$ indicates a correct answer on trial $i$, $R_i \in \{0,1\}$). The key modeling assumption is that each trial is an independent Bernoulli draw with unknown success probability $p$. A conjugate Beta prior, $\mathrm{Beta}(\alpha_1^0, \alpha_0^0)$, is placed over $p$. With a uniform prior, $\alpha_1^0 = \alpha_0^0 = 1$, Bayesian updating yields the posterior:

$$p \mid \{C,N\} \sim \mathrm{Beta}(1 + C,\, 1 + N - C)$$

The posterior mean (i.e., the scaled Pass@1 score) is

$$\mu = \mathbb{E}[p \mid C, N] = \frac{C + 1}{N + 2}$$

By construction, this posterior mean is a strictly increasing function of the empirical proportion $C/N$, ensuring that rankings according to scaled Pass@1 or raw Pass@1 are order-equivalent, but the Bayesian estimator enjoys better stability—particularly in small-$N$ regimes or when $C=0$ or $C=N$, where $\mu$ avoids degeneration to the pathological values 0 and 1 (Hariri et al., 5 Oct 2025).

Accompanying $\mu$ are credible intervals—analytically tractable as quantiles of the Beta posterior—which provide explicit uncertainty intervals for interpretability and statistically principled model comparisons.

2. Motivation and Stability in Low-data Regimes

A central motivation for scaled Pass@1 is the instability of raw Pass@1 as a function of $C$ and $N$, especially with limited compute or rare events. When $N$ is small, the observed success rate $C/N$ can yield misleadingly extreme or binary conclusions (e.g., zero successes in five trials yields Pass@1 = 0)—conferring neither practical insight nor robust statistical inference.

Scaled Pass@1 alleviates this by shrinking estimates away from the boundaries. For instance, with $N=5$ and $C=0$, scaled Pass@1 $\mu = 1/7 \approx 0.143$, with a wide 95% credible interval, contrasting with the plug-in estimate of 0. As $N$ increases, the smoothing effect vanishes; for $N=100$, $C=50$, scaled Pass@1 and raw Pass@1 both approximate 0.5. This shrinkage effect prevents overconfident reporting and allows uncertainty quantification that reflects actual informational content in the data (Hariri et al., 5 Oct 2025, Kazdan et al., 6 Oct 2025).

3. Computation and Implementation Details

The computation of scaled Pass@1 proceeds as follows:

• Prior setup: Set uniform prior hyperparameters $\alpha^0_1=\alpha^0_0=1$ (or other prior if prior knowledge warrants).
• Posterior update: Compute $\alpha_1 = 1 + C$, $\alpha_0 = 1 + (N - C)$.
• Score computation: Posterior mean is $\mu = \alpha_1/(\alpha_1+\alpha_0)$.
• Credible intervals: The central $100 \cdot (1-\delta)\%$ credible interval is given by the inverse regularized incomplete beta function:
  • Lower bound $$L = \mathrm{Beta}^{-1}(\delta/2; \alpha_1, \alpha_0)$$
  • Upper bound $$U = \mathrm{Beta}^{-1}(1 - \delta/2; \alpha_1, \alpha_0)$$

This workflow yields both a robust point estimate and associated interval, directly facilitating model ranking and significance assessment through non-overlapping intervals.

4. Scaling to Pass@$k$ and Gradient Structure

The Pass@1 metric is the $k=1$ case of Pass@$k$, the probability of at least one successful outcome in $k$ independent samples. The scaled Pass@1 score serves as the canonical, robust estimator for the base success probability, from which Pass@$k$ for any $k$ can be derived analytically if $p$ were known:

$\mathrm{Pass}@k = 1 - (1-p)^k$

When $p$ is estimated by $\mu$, plug-in estimators remain susceptible to the same issues as with $k=1$. Bayesian estimation remedies this via closed-form expressions for the probability of at least one success in $K$ future samples using the beta-binomial model:

$$\mathbb{E}[\mathrm{Pass}@K \mid C, N] = 1 - \frac{B(\alpha_1, \alpha_0 + K)}{B(\alpha_1, \alpha_0)}$$

where $B$ is the beta function (Kazdan et al., 6 Oct 2025).

Moreover, in the context of reinforcement learning and policy gradients, Pass@$k$ gradients decompose into a positive per-example reweighting of the Pass@1 gradient: $$\nabla_\theta J_{k}(x) = k(1-J_{1}(x))^{k-1} \nabla_\theta J_{1}(x)$$, establishing a tight algorithmic and statistical link between Pass@$k$ objectives and the Pass@1 landscape (Yu, 20 Nov 2025, Barakat et al., 24 Feb 2026).

5. Practical Applications, Limitations, and Guidelines

Scaled Pass@1 is now recommended as the primary evaluation metric for LLM binary tasks in compute-constrained or low-sample settings, superseding raw Pass@1 and ad hoc estimators. Typical applications include evaluating reasoning and code-generation LLMs, especially on contests or datasets with heterogeneous item difficulty and limited validation queries.

Empirical work (Hariri et al., 5 Oct 2025) demonstrates that scaled Pass@1 leads to faster estimate convergence and increased stability in model rankings versus Pass@$k$, avg@$N$, or bootstrapped plug-in metrics. The explicit uncertainty quantification enables a rigorous rule for distinguishing statistically significant differences (non-overlapping credible intervals), clarifying when apparent performance gaps are genuine or just statistical noise.

However, over-aggregation or global proxies can miss prompt-level heterogeneity that true Pass@$k$ metrics capture, and neglect of prompt-wise interference can yield misleading learning dynamics during policy optimization (Barakat et al., 24 Feb 2026). For full robustness, credible intervals and prompt-wise analyses should be reported, and per-datum scaling may be warranted when response distributions are highly non-uniform.

6. Broader Implications and Relation to Other Metrics

Adoption of scaled Pass@1 provides a unified, Bayesian framework for both binary (Pass@1) and multi-sample (Pass@$k$) evaluation. The same beta-binomial machinery yields robust extrapolations for extreme-$k$ settings (e.g., forecasting rare jailbreak events with few observed successes).

Its order-equivalence with average accuracy aids interpretability while eliminating pathologies inherent in deterministic plug-in estimators. Credible intervals are fully closed-form, simplifying reporting workflows.

Researchers have found that direct optimization of Pass@$k$ can degrade Pass@1 due to gradient weighting favoring low-success prompts that negatively interfere with population-level accuracy. Scaled Pass@1, with its Bayesian shrinkage and uncertainty control, partially mitigates these issues but does not replace the need for careful, context-aware policy design and evaluation (Yu, 20 Nov 2025, Barakat et al., 24 Feb 2026).

7. Summary Table: Scaled Pass@1 versus Related Metrics

Metric	Formula	Strengths
Raw Pass@1	$C/N$	Simple, interpretable; matches empirical accuracy for large $N$
Scaled Pass@1	$(C+1)/(N+2)$	Stable for small $N$; avoids pathologies; credible intervals
Plug-in Pass@$k$	$1 - (1 - C/N)^k$	Simple extrapolation; unstable for $C=0$ or $C=N$
Bayesian Pass@$k$	$1 - \frac{B(C+1,N-C+1+K)}{B(C+1,N-C+1)}$	Robust, uncertainty-aware, can handle $K \gg N$

All Bayesian estimators above use the Beta(1,1) (uniform) prior unless otherwise specified. For models where prior knowledge is warranted, hyperparameters can be tuned (e.g., via empirical Bayes or cross-validation).

---

Scaled Pass@1 constitutes a statistically principled, practically robust approach to LLM evaluation and ranking, unifying binary success metrics with their multi-sample generalizations, and providing explicit, interpretable measures of uncertainty under computational constraints (Hariri et al., 5 Oct 2025, Kazdan et al., 6 Oct 2025, Yu, 20 Nov 2025, Barakat et al., 24 Feb 2026).