Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.

• The paper introduces PRECISE, a prediction-powered estimator that combines a small human-labeled set with large-scale LLM judgments to produce unbiased, lower-variance ranking metrics despite judge errors.
• The method reduced Precision@4 standard error by 21% on ESCI with 30 human labels and 60,000 LLM judgments, while Haiku achieved 0.29 points of bias at $79 compared with 0.70 points at $946 for Sonnet.
• The production evaluation used 100 human labels and 8,400 LLM judgments to correctly predict the ranking of search variants, including T1’s subsequent gains of 407 basis points in sales and 571 basis points in click-through rate.

This paper extends Prediction-Powered Inference (PPI) to ranking evaluation metrics, presenting a framework (PRECISE) that combines a small set of human-labeled queries with a large LLM-judged set to produce bias-corrected, low-variance estimates of metrics such as Precision@K. The central claim is that the resulting estimator is provably unbiased regardless of the LLM judge's error profile, and that this statistical guarantee can be operationalized at production scale with minimal annotation cost.

Motivation and positioning

Human evaluation of retrieval systems is expensive, and small labeled sets yield confidence intervals too wide to distinguish real system improvements from sampling noise. LLM-as-a-Judge approaches reduce annotation cost but introduce systematic biases that distort evaluation metrics when used as substitutes for human annotation. Rather than pursuing the common strategy of building better judges through prompt engineering, fine-tuning, or multi-agent debate, the paper takes a complementary stance: accept the judge's bias and correct for it statistically.

The framework builds on PPI and PPI++ (Angelopoulos et al., 2023), a semi-supervised estimation method. The estimator takes the form

$$\hat{\mu}_{\text{PPI}} = \frac{\lambda}{N}\sum_{i=1}^{N} \tilde{\mu}_u^{(i)} + \frac{1}{n}\sum_{i=1}^{n}\Big[\phi_i - \lambda\, \tilde{\mu}_g^{(i)}\Big],$$

where the first term is the LLM-based estimate over the large unlabeled set $\mathcal{D}_u$ ($N \gg n$) and the second term measures the judge's deviation from human ground truth on the gold set $\mathcal{D}_g$ and subtracts it. The tuning parameter $\lambda \in [0,1]$ is chosen to minimize variance; the estimator remains unbiased for any $\lambda > 0$. When the judge is well-calibrated, $\lambda \simeq 1$ lets the full unlabeled set drive variance reduction; when it is heavily biased, shrinking $\lambda$ toward 0 reweights the estimator toward gold labels. The key property is that additional LLM-judged examples reduce variance without introducing new bias.

Handling hierarchical metrics

A technical obstacle for ranking metrics is granularity mismatch: for Precision@K, human annotations are per query-document pair, but the metric is per-query. Naively, the PPI output space would be $\{0,1\}^{|C|}$ over all corpus documents—computationally intractable when $|C|$ is in the millions.

The paper resolves this by observing that Precision@K depends only on the top-$\mathcal{D}_u$0 retrieved documents, so the metric reduces to a scaled dot product over sparse vectors, $\mathcal{D}_u$1. All non-retrieved documents collapse into a single weight on the all-zero $\mathcal{D}_u$2-vector, shrinking the output space to $\mathcal{D}_u$3. The LLM judge supplies per-document relevance probabilities $\mathcal{D}_u$4 for the $\mathcal{D}_u$5-th ranked result, and a joint distribution over the $\mathcal{D}_u$6-length binary vectors is formed under a conditional independence assumption:

$\mathcal{D}_u$7

The LLM-based estimates are then computed by summing $\mathcal{D}_u$8 over all $\mathcal{D}_u$9 vectors, which is tractable for typical $N \gg n$0. The authors state that this hierarchical formulation applies in principle to any metric aggregating fine-grained judgments (e.g., per-claim factuality, per-turn dialogue quality), though only Precision@K is empirically validated.

Empirical results

On the ESCI shopping-retrieval benchmark (Reddy et al., 2022), with Claude 3 Sonnet and Haiku as judges, $N \gg n$1 gold annotations, and $N \gg n$2 unlabeled queries, the results for Precision@4 estimation are:

Estimator	Bias (pts)	Std. Err. (pts)	Cost
Gold only ($N \gg n$3)	1.04	4.45	—
PPI + Claude 3 Sonnet	0.70	3.50	$946
PPI + Claude 3 Haiku	0.29	3.86	$79

Sonnet reduces the standard error from 4.45 to 3.50, a 21% relative reduction, while keeping bias low. Haiku achieves the lowest bias (0.29) at 12× lower inference cost, indicating that the bias-correction mechanism does not require the most capable judge. The sampling distributions show the PPI estimator is both narrower than gold-only and centered on the true value, whereas LLM-only estimates are visibly biased.

A notable scaling finding is that the framework plateaus at a 100× unlabeled-to-gold ratio: $N \gg n$4 LLM judgments yield nearly identical standard error to $N \gg n$5 when $N \gg n$6. This implies that the practical annotation budget is dominated by the gold set size rather than the LLM-judged volume, which materially lowers the cost of deployment.

Production validation

In a production search system, the framework ranked three system variants (Control, T1, T2) using 100 human labels and 8,400 LLM judgments, produced with 2 hours of domain-expert annotation. The predicted ranking (T1 > T2 > Control) was confirmed by A/B testing, with T1 delivering +407 basis points in daily sales and +571 bps in click-through rate. Without the PPI correction, LLM-only estimates could not distinguish between variants, because systematic upward bias inflated all estimates similarly; the semi-supervised correction restored discriminative power. This is the paper's strongest practical claim: a statistically grounded offline evaluation matched a live experiment at a small fraction of the annotation cost.

Limitations and open questions

The paper concedes three limitations. First, the hierarchical PPI extension is validated only on Precision@K for retrieval; generalization to other hierarchical metrics such as per-claim factuality or per-turn dialogue quality remains untested. Second, the conditional independence assumption in the joint distribution over top-$N \gg n$7 relevance vectors may fail when document relevance depends on the presence of other documents, as in diversity-sensitive ranking; relaxing this assumption is left open. Third, the framework requires a gold set drawn from the same distribution as the unlabeled set—temporal drift or other distribution shift could degrade the bias correction. The paper also notes that the method reduces, but does not eliminate, the need for human annotation. Future directions identified include incorporating synthetic covariates from LLM-generated datasets, connecting to doubly robust estimation for online evaluation, aggregating multiple judges via multi-objective optimization rather than a single all-purpose prompt (which the authors report is brittle), and extending the approach to produce calibrated critic signals for agentic systems.

Conclusion

The paper demonstrates that PPI can be adapted to ranking evaluation, including hierarchical metrics whose granularity differs from that of the underlying annotations, by exploiting the sparsity of top-$N \gg n$8 relevance. Empirically, 30 human labels plus LLM judgments yield a 21% standard-error reduction on ESCI, and a 100-label production evaluation correctly predicted A/B-test outcomes (+407 bps in daily sales) that LLM-only estimates could not resolve. The main open questions concern generalization beyond Precision@K, the conditional independence assumption, and robustness to distribution shift between gold and unlabeled sets.