PromptEval Estimator for LLM Performance

• PromptEval Estimator is a formal method that quantifies LLM performance distributions over diverse prompts by capturing prompt sensitivity with balanced sampling.
• It employs a logistic item response theory model with conditional expectation-based imputation to efficiently estimate prompt-level accuracies from limited data.
• The approach yields robust quantile estimates and risk-sensitive metrics, supporting improved model comparison and decision making under constrained evaluation budgets.

A PromptEval Estimator is a formal procedure to quantify the performance of LLMs over distributions of prompts, capturing prompt sensitivity and robustly estimating performance distributions, quantiles, and statistics under limited evaluation budgets. The methodology draws from item response theory, stratified experimental design, and conditional expectation estimators to consistently recover the full distribution of prompt-level accuracies using only a small subset of model executions. This approach is central to recent works on robust LLM evaluation, replacing single-prompt leaderboards with empirical distributions that support risk-sensitive decision making and comparison (Polo et al., 2024).

1. Problem Formulation and Statistical Model

PromptEval Estimator, as formalized in "Efficient multi-prompt evaluation of LLMs" (Polo et al., 2024), addresses the problem of estimating the distribution of LLM performance across many prompt templates. Let $J$ denote the number of prompt templates and $N$ the number of held-out task examples. For each prompt–example pair $(j, n)$, the binary outcome $Y_{jn} \in \{0,1\}$ records model correctness, with $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$ being the average accuracy of prompt $j$. The global objective is to estimate the empirical distribution $F(x) = \frac{1}{J}\sum_{j=1}^J 1_{[p_j \geq x]}$ and quantile function $Q(p) = \inf\{x: F(x) \geq p\}$, such as the median or 95th percentile prompt performance.

Given the prohibitive cost of evaluating all $J \times N$ pairs, only a subset $E \subseteq \{1,\ldots,J\} \times \{1,\ldots,N\}$ of size $N$0 is observed. The estimator employs a two-way balanced (stratified) sampling strategy to guarantee that each prompt, as well as each task example, receives a roughly equal share of evaluations.

The underlying statistical model is a logistic IRT (Rasch or extended) model:

$N$1

where $N$2 and $N$3 represent the prompt "easiness" and example "difficulty," respectively.

2. Conditional Expectation Estimator Construction

The PromptEval estimator computes a conditional expectation for each $N$4 using observed data and parametric imputation:

$N$5

where:

• $N$6, $N$7
• $N$8 are fitted via penalized maximum likelihood (Bernoulli log-likelihood) over observed data
• $N$9 are optional prompt and example covariates for higher expressiveness (embeddings or handcrafted features)

This estimator linearly interpolates between empirical means for observed $(j, n)$0 pairs and model-based imputation for unobserved cases, leading to consistent recovery of the true $(j, n)$1 as $(j, n)$2 under mild regularity.

3. Efficient Sampling and Implementation

Two-way balanced sampling (Algorithm 2 in (Polo et al., 2024)) ensures that the allocation of evaluation budget is uniform across prompts and examples, stabilizing the estimation even when only a small fraction ($(j, n)$3) of all possible pairs is observed. The estimation procedure consists of:

• Fitting the low-dimensional logistic model on observed entries ($(j, n)$4 for $(j, n)$5-dimensional covariates)
• Inference of all missing $(j, n)$6 via the model, with vectorized computation and GPU-accelerated summation
• Formation of the estimated CDF $(j, n)$7 and quantiles $(j, n)$8 by sorting the $(j, n)$9

This design yields scalable inference procedures applicable for $Y_{jn} \in \{0,1\}$0 up to several hundreds with budgets as low as $Y_{jn} \in \{0,1\}$1–$Y_{jn} \in \{0,1\}$2 (equivalent to just a few full-prompt runs).

4. Theoretical Guarantees and Error Analysis

Under bounded covariates and sufficient unseen entries per prompt (i.e., $Y_{jn} \in \{0,1\}$3), the PromptEval estimator is statistically consistent:

• $Y_{jn} \in \{0,1\}$4
• $Y_{jn} \in \{0,1\}$5 for all $Y_{jn} \in \{0,1\}$6
• The Wasserstein-1 distance $Y_{jn} \in \{0,1\}$7

Concentration is driven by a combination of high-probability bounds from Hoeffding's inequality (on observed entries) and the Lipschitz continuity of the logistic function for modeled entries. Bootstrap procedures over the sampled evaluation pairs support empirical confidence intervals for quantiles and other statistics.

5. Robust Performance Metrics and Applications

PromptEval outputs enable robust, risk-sensitive LLM benchmarking:

• Quantiles: Median ($Y_{jn} \in \{0,1\}$8), 95th percentile ($Y_{jn} \in \{0,1\}$9), and 5th percentile ($p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$0) reflect performance suitable for expert, typical, and worst-case prompt engineering contexts
• Performance distribution visualization: Empirical CDFs reveal prompt sensitivity at a glance
• Comparative model assessment: Stochastic dominance (all quantiles better) supports robust model ranking, as opposed to noise-prone single-prompt scoring
• Best-prompt identification: Budget-constrained "best-arm" algorithms can efficiently locate the most effective prompt, as demonstrated on MMLU, BBH, and LMentry

Empirical findings indicate that PromptEval with covariate-rich variants can estimate median prompt accuracy to $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$1 error with budgets of $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$2 and extreme quantiles to $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$3 error with $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$4–$p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$5, outperforming naive averaging by factors of $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$6–$p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$7 (Polo et al., 2024).

6. Model Assumptions, Limitations, and Extensions

The assumption of a low-dimensional IRT/logistic model is appropriate when prompt–example interactions are well-approximated by independent prompt and example effects. If this assumption is violated—e.g., if specific prompt–example pairs exhibit interaction effects not reducible to additive terms—estimator residuals may indicate model misspecification.

Practical considerations include computing costs for large $p_j = S_j = \frac{1}{N}\sum_{n=1}^N Y_{jn}$8 (addressed with vectorized computations and GPU parallelism) and the potential need for richer covariates (embeddings, hand-crafted features) to capture prompt structure. Regularization and model selection mechanisms can be applied to select between Rasch, discrete, and embedding-based PromptEval variants. Bootstrap confidence intervals provide empirical uncertainty quantification.

A plausible implication is that the PromptEval Estimator can serve as a foundation for more general prompt performance inference frameworks, such as those supporting active prompt selection, risk-aware deployment, and ensemble model evaluation, as suggested in related works (Polo et al., 2024).

7. Relevance and Impact for LLM Evaluation

PromptEval Estimator marks a methodological advance in LLM benchmarking by shifting focus from single-prompt or static leaderboard reporting to performance distributions that respect inherent prompt sensitivity. Its design is compatible with applications including LLM-as-a-judge, robust model selection under uncertain prompt conditions, and automated risk analysis for safety-critical or high-reliability contexts.

PromptEval is model-agnostic and directly applicable across diverse tasks and benchmarks. Its thoroughly validated empirical and theoretical properties make it a practical drop-in for reproducible, risk-sensitive evaluation pipelines in contemporary LLM research (Polo et al., 2024).