--- title: 'Pass@$k$ Inference: LLM Diversity Evaluation' url: https://www.emergentmind.com/topics/pass-k-inference type: topic --- # Pass@$k$ Inference: LLM Diversity Evaluation Pass@$k$ inference is a multi-sample evaluation metric and inference protocol central to the empirical assessment and deployment of large language models (LLMs) in verifiable reasoning tasks such as mathematical problem solving, code synthesis, and formal logic. It quantifies the probability that at least one correct solution is obtained in $k$ independent samples drawn from a model, thereby serving as a practical measure of diversity-driven coverage. Despite its intuitive appeal and widespread adoption as an evaluation standard, the Pass@$k$ metric exhibits nuanced properties as both a diagnostic and an objective for reinforcement learning with verifiable rewards (RLVR), critically shaping inference scaling and exploration strategies. ## 1. Formal Definition and Mathematical Properties For an input $x$ (e.g., a prompt), let $\pi_\theta(y | x)$ denote an LLM’s generative policy and $V(x, y)\in\{0,1\}$ be a binary correctness verifier. The single-sample (“Pass@$1$”) success probability is: \[ J_1(x;\theta) = \Pr_{y\sim\pi_\theta(\cdot|x)} \left[ V(x, y) = 1 \right] = \sum_{y} \pi_\theta(y|x) V(x, y). \] The $k$-sample Pass@$k$ metric gives the probability that at least one sample in $k$ independent draws is correct: \[ J_k(x;\theta) = 1 - (1 - J_1(x;\theta))^k = \mathbb{E}_{y_1,...,y_k\sim\pi_\theta}\left[1 - \prod_{i=1}^k (1 - V(x, y_i))\right]. \] For a batch of $n$ samples ($n\geq k$) with $c$ correct, an unbiased estimator for empirical evaluation is: \[ \mathrm{Pass}@k := 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}. \] Pass@$k$ is therefore interpretable as the Bernoulli coverage probability over $k$ attempts and is analytically tractable under i.i.d. sampling. ## 2. Policy Gradient Structure and Learning Dynamics The gradient of $J_k$ with respect to $\theta$ follows by the chain rule: \[ \nabla_\theta J_k(x;\theta) = k(1-J_1(x;\theta))^{k-1} \nabla_\theta J_1(x;\theta). \] Where, \[ \nabla_\theta J_1(x;\theta) = \mathbb{E}_{y\sim\pi_\theta} \left[ V(x,y)\nabla_\theta\log\pi_\theta(y|x) \right]. \] This decomposition shows that $\nabla_\theta J_k$ is a positive scalar reweighting $\alpha_k(x;\theta)$ of the Pass@$1$ gradient: \[ \alpha_k(x;\theta) = k(1-J_1(x;\theta))^{k-1}. \] Key regimes: - When $J_1\approx0$, $\alpha_k\approx k$ but sampling-based gradient estimates are vacuous (rarely any correct $V=1$ samples). - When $J_1\approx1$, $\alpha_k\approx 0$, yielding a vanishing learning signal. Thus, Pass@$k$ gradients vanish in both the high-failure (exploration) and high-success regimes, failing to provide meaningful learning incentives at the boundaries [2511.16231]. ## 3. Exploration Collapse and Prompt Interference Phenomena Over repeated training with policy gradients (e.g., RLVR), modes discovered by the policy (with significant probability mass) are reinforced, while undiscovered modes (with probability $\epsilon$) are exponentially unlikely to be found in $k$ samples ($\sim k\epsilon$). This "exploration collapse" leads to: \[ \lim_{t\to\infty} \Delta_t(k) = J_k - J_1 = 1 - (1-p_t)^k - p_t \to 0. \] Thus, as the model becomes confident on one mode, Pass@$k$ converges to Pass@$1$, eliminating multi-sample benefits. Additionally, direct Pass@$k$ optimization can degrade Pass@$1$ due to "prompt interference" [2602.21189]: Pass@$k$ reweights gradient contributions toward “hard” prompts with low $J_1$, but these can be negatively aligned with Pass@$1$. The expected gradient inner product \[ \langle\nabla J_k, \nabla J_1\rangle = \mathbb{E}_x[w_k(p_\theta(x)) a_\theta(x)], \] can be negative, guaranteeing that improving Pass@$k$ can reduce Pass@$1$ if negatively interfering prompts dominate. Empirically, strong negative alignment is observed for hard examples in mathematical reasoning benchmarks. ## 4. Pass@$k$ as Practical Diagnostic vs. Optimization Objective Pass@$k$ is essential as a diagnostic to gauge uncoverable solution diversity at inference: a significant gap $J_k - J_1$ in held-out evaluation indicates the model has rare, correct modes accessible via sampling. However, due to collinearity and vanishing learning signal in the gradient structure, direct optimization of Pass@$k$ provides no exploration benefit over Pass@$1$ and can collapse diversity [2511.16231]. Instead, RL objectives promoting entropy or coverage, explicit diversity bonuses, or advantage-shaping with surrogate rewards are preferred for exploration. Table: Summary of Pass@$k$ Metric’s Roles | Role | Supported? | Mechanism | |------------------|---------------|----------------------------------------------------------| | Diagnostic tool | Yes | Measures coverage of rare correct solutions | | Training target | No (generally)| Gradient vanishes/extremely collinear; harms exploration | | Exploration Aid | No | Collapses to Pass@$1$ as mode confidence rises | | Reliability Tune | Indirect | Adjust $k$ at inference for desired coverage/reliability | ## 5. Recommendations, Limitations, and Best Practices - **Training:** Optimize for coverage/exploration using entropy-driven objectives or explicit diversity-enhancement, not Pass@$k$ directly. Monitor Pass@$k$ and Pass@$1$ side-by-side on held-out datasets. - **Inference:** Use Pass@$k$ to decide the number of samples ($k$) that meaningfully increase coverage. A diminishing gap between Pass@$k$ and Pass@$1$ indicates "saturation" in exploration. - **Sampling Strategies:** Increase temperature, top-$p$, or top-$k$ to boost diversity—but excessive randomness will decrease Pass@$1$. - **$k$ Selection:** Choose the minimal $k$ that stabilizes diversity for cost-efficient inference, e.g., $k=5$–$20$ for complex math, $k=50$–$100$ for code synthesis, balancing latency constraints. - **Interpretation:** In small-discrete-output regimes, large $k$ Pass@$k$ degenerates towards the fraction of tasks with nonzero success probability, conflating guessing with reasoning [2510.08325]. Complement Pass@$k$ with coverage-versus-reliability metrics (e.g., Cover@$τ$). ## 6. Empirical and Theoretical Implications Empirical evaluations confirm that Pass@$k$ is highly sensitive to sampling protocol and underlying policy concentration. On real tasks, reinforced over-concentration (e.g., via RLVR) increases Pass@$1$ at the expense of diversity and Pass@$k$. The trade-off is robust to architectures and datasets, and the theoretical gradient structure predicts this consistently [2511.16231, 2602.21189]. Best-practice is therefore to use Pass@$k$ for inference-time exploration auditing and to optimize with objectives that reward spread or coverage explicitly. ## 7. Extensions and Future Directions Recent research proposes more reliable evaluation protocols such as Bayesian posterior estimation [2510.04265], breadth-depth metrics (Cover@$τ$) [2510.08325], and scaling laws for Pass@$k$ as a function of cost, task hardness, and number of attempts [2410.16377, 2510.05197]. These alternatives expose the limitations of Pass@$k$-centric evaluation for fine-grained model comparison, especially in high-variance, low-coverage, or discrete-output domains. Open directions include adaptive sampling strategies, variance reduction in Pass@$k$ estimators, and metrics that jointly capture diversity and reliability. --- In summary, Pass@$k$ inference is a key multi-sample success metric characterizing coverage in verifiable LLM tasks, with robust analytic properties and well-understood limitations. While indispensable as a diagnostic of latent diversity, it is ill-suited as a direct RL objective and susceptible to collapse when used naively for optimization. Research consensus is to deploy Pass@$k$ as a principled, inference-time measurement and to pair it with explorationally-motivated objectives and complementary reliability metrics in modern LLM evaluation pipelines [2511.16231, 2602.21189, 2510.08325, 2510.04265]. Source: https://www.emergentmind.com/topics/pass-k-inference