--- title: Scaled Pass@1 Score in LLM Evaluation url: https://www.emergentmind.com/topics/scaled-pass-1-score type: topic --- # Scaled Pass@1 Score in LLM Evaluation A scaled Pass@1 score is a statistically principled metric for evaluating the single-sample success probability of large language models (LLMs) on binary verifiable tasks. Unlike raw Pass@1—which is simply the observed proportion of correct responses in $N$ trials—scaled Pass@1 adopts a Bayesian approach, combining observed data with a conjugate Beta prior to provide robust point estimates, controlled shrinkage in low-sample regimes, and credible intervals for model comparison. The metric generalizes naturally to other Pass@$k$-type metrics, provides a unified framework for ranking and reporting, and addresses several known pitfalls of naïve or plug-in estimators when evaluating LLM performance. ## 1. Formal Definition and Bayesian Framework Let $N$ denote the number of independent evaluation trials, and $C$ the number of successful outcomes ($R_i = 1$ indicates a correct answer on trial $i$, $R_i \in \{0,1\}$). The key modeling assumption is that each trial is an independent Bernoulli draw with unknown success probability $p$. A conjugate Beta prior, $\mathrm{Beta}(\alpha_1^0, \alpha_0^0)$, is placed over $p$. With a uniform prior, $\alpha_1^0 = \alpha_0^0 = 1$, Bayesian updating yields the posterior: \[ p \mid \{C,N\} \sim \mathrm{Beta}(1 + C,\, 1 + N - C) \] The posterior mean (i.e., the scaled Pass@1 score) is \[ \mu = \mathbb{E}[p \mid C, N] = \frac{C + 1}{N + 2} \] By construction, this posterior mean is a strictly increasing function of the empirical proportion $C/N$, ensuring that rankings according to scaled Pass@1 or raw Pass@1 are order-equivalent, but the Bayesian estimator enjoys better stability—particularly in small-$N$ regimes or when $C=0$ or $C=N$, where $\mu$ avoids degeneration to the pathological values 0 and 1 [2510.04265]. Accompanying $\mu$ are credible intervals—analytically tractable as quantiles of the Beta posterior—which provide explicit uncertainty intervals for interpretability and statistically principled model comparisons. ## 2. Motivation and Stability in Low-data Regimes A central motivation for scaled Pass@1 is the instability of raw Pass@1 as a function of $C$ and $N$, especially with limited compute or rare events. When $N$ is small, the observed success rate $C/N$ can yield misleadingly extreme or binary conclusions (e.g., zero successes in five trials yields Pass@1 = 0)—conferring neither practical insight nor robust statistical inference. Scaled Pass@1 alleviates this by shrinking estimates away from the boundaries. For instance, with $N=5$ and $C=0$, scaled Pass@1 $\mu = 1/7 \approx 0.143$, with a wide 95% credible interval, contrasting with the plug-in estimate of 0. As $N$ increases, the smoothing effect vanishes; for $N=100$, $C=50$, scaled Pass@1 and raw Pass@1 both approximate 0.5. This shrinkage effect prevents overconfident reporting and allows uncertainty quantification that reflects actual informational content in the data [2510.04265, 2510.05197]. ## 3. Computation and Implementation Details The computation of scaled Pass@1 proceeds as follows: - **Prior setup:** Set uniform prior hyperparameters $\alpha^0_1=\alpha^0_0=1$ (or other prior if prior knowledge warrants). - **Posterior update:** Compute $\alpha_1 = 1 + C$, $\alpha_0 = 1 + (N - C)$. - **Score computation:** Posterior mean is $\mu = \alpha_1/(\alpha_1+\alpha_0)$. - **Credible intervals:** The central $100 \cdot (1-\delta)\%$ credible interval is given by the inverse regularized incomplete beta function: - Lower bound $L = \mathrm{Beta}^{-1}(\delta/2; \alpha_1, \alpha_0)$ - Upper bound $U = \mathrm{Beta}^{-1}(1 - \delta/2; \alpha_1, \alpha_0)$ This workflow yields both a robust point estimate and associated interval, directly facilitating model ranking and significance assessment through non-overlapping intervals. ## 4. Scaling to Pass@$k$ and Gradient Structure The Pass@1 metric is the $k=1$ case of Pass@$k$, the probability of at least one successful outcome in $k$ independent samples. The scaled Pass@1 score serves as the canonical, robust estimator for the base success probability, from which Pass@$k$ for any $k$ can be derived analytically if $p$ were known: \[ \mathrm{Pass}@k = 1 - (1-p)^k \] When $p$ is estimated by $\mu$, plug-in estimators remain susceptible to the same issues as with $k=1$. Bayesian estimation remedies this via closed-form expressions for the probability of at least one success in $K$ future samples using the beta-binomial model: \[ \mathbb{E}[\mathrm{Pass}@K \mid C, N] = 1 - \frac{B(\alpha_1, \alpha_0 + K)}{B(\alpha_1, \alpha_0)} \] where $B$ is the beta function [2510.05197]. Moreover, in the context of reinforcement learning and policy gradients, Pass@$k$ gradients decompose into a positive per-example reweighting of the Pass@1 gradient: $\nabla_\theta J_{k}(x) = k(1-J_{1}(x))^{k-1} \nabla_\theta J_{1}(x)$, establishing a tight algorithmic and statistical link between Pass@$k$ objectives and the Pass@1 landscape [2511.16231, 2602.21189]. ## 5. Practical Applications, Limitations, and Guidelines Scaled Pass@1 is now recommended as the primary evaluation metric for LLM binary tasks in compute-constrained or low-sample settings, superseding raw Pass@1 and ad hoc estimators. Typical applications include evaluating reasoning and code-generation LLMs, especially on contests or datasets with heterogeneous item difficulty and limited validation queries. Empirical work [2510.04265] demonstrates that scaled Pass@1 leads to faster estimate convergence and increased stability in model rankings versus Pass@$k$, avg@$N$, or bootstrapped plug-in metrics. The explicit uncertainty quantification enables a rigorous rule for distinguishing statistically significant differences (non-overlapping credible intervals), clarifying when apparent performance gaps are genuine or just statistical noise. However, over-aggregation or global proxies can miss prompt-level heterogeneity that true Pass@$k$ metrics capture, and neglect of prompt-wise interference can yield misleading learning dynamics during policy optimization [2602.21189]. For full robustness, credible intervals and prompt-wise analyses should be reported, and per-datum scaling may be warranted when response distributions are highly non-uniform. ## 6. Broader Implications and Relation to Other Metrics Adoption of scaled Pass@1 provides a unified, Bayesian framework for both binary (Pass@1) and multi-sample (Pass@$k$) evaluation. The same beta-binomial machinery yields robust extrapolations for extreme-$k$ settings (e.g., forecasting rare jailbreak events with few observed successes). Its order-equivalence with average accuracy aids interpretability while eliminating pathologies inherent in deterministic plug-in estimators. Credible intervals are fully closed-form, simplifying reporting workflows. Researchers have found that direct optimization of Pass@$k$ can degrade Pass@1 due to gradient weighting favoring low-success prompts that negatively interfere with population-level accuracy. Scaled Pass@1, with its Bayesian shrinkage and uncertainty control, partially mitigates these issues but does not replace the need for careful, context-aware policy design and evaluation [2511.16231, 2602.21189]. ## 7. Summary Table: Scaled Pass@1 versus Related Metrics | Metric | Formula | Strengths | |--------------------|-----------------------------------|---------------------| | Raw Pass@1 | $C/N$ | Simple, interpretable; matches empirical accuracy for large $N$ | | Scaled Pass@1 | $(C+1)/(N+2)$ | Stable for small $N$; avoids pathologies; credible intervals | | Plug-in Pass@$k$ | $1 - (1 - C/N)^k$ | Simple extrapolation; unstable for $C=0$ or $C=N$ | | Bayesian Pass@$k$ | $1 - \frac{B(C+1,N-C+1+K)}{B(C+1,N-C+1)}$ | Robust, uncertainty-aware, can handle $K \gg N$ | All Bayesian estimators above use the Beta(1,1) (uniform) prior unless otherwise specified. For models where prior knowledge is warranted, hyperparameters can be tuned (e.g., via empirical Bayes or cross-validation). --- Scaled Pass@1 constitutes a statistically principled, practically robust approach to LLM evaluation and ranking, unifying binary success metrics with their multi-sample generalizations, and providing explicit, interpretable measures of uncertainty under computational constraints [2510.04265, 2510.05197, 2511.16231, 2602.21189]. Source: https://www.emergentmind.com/topics/scaled-pass-1-score