CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

Abstract: Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an $M \times N$ matrix of evaluation scores, where $M$ is the total number of models and $N$ is the total number of evaluation prompts. We assume that a subset of these $M$ models are targeted for evaluation. For these target models only a small fraction, $p$, of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels $p$, we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.

• The paper introduces a novel matrix completion framework that reframes model evaluation, significantly reducing label acquisition and inference costs.
• The paper employs CrossPPI to correct prediction bias, achieving up to a 30% reduction in confidence interval width with sparse data.
• The paper validates its approach across diverse benchmarks, showing that up to 40–60% fewer direct labels are needed for comparable estimation variance.

CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

Overview

"CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion" (2607.05046) proposes a paradigm for evaluating generative AI models that leverages historical evaluation data and inherent inter-model correlations to significantly improve the statistical efficiency of model performance estimation. The approach is grounded in reframing model evaluation as a matrix completion problem, utilizing cross-prediction-powered inference (CrossPPI) to rigorously control estimation bias. Substantial empirical efficiency gains are demonstrated across diverse evaluation benchmarks.

Motivation and Problem Formulation

Large-scale generative model evaluation is expensive due to the need for both substantial label acquisition and significant model inference costs. While some prior works have reduced annotation costs with weaker automatic raters or active sampling, these methods generally fail to exploit correlations found in repeated evaluations among closely related model variants.

The core insight of CollabEval is that model performance scores, collected across evaluation datasets and model variants, exhibit strong low-rank structure. This enables accurate imputation of missing scores using historical data, thus reducing the necessity for dense labeling of novel models. Evaluations are formalized as an $M\times N$ partially observed score matrix over $M$ models and $N$ prompts, with dense historical data (“anchor models”) and sparse labeled samples on “target models”.

Figure 1: CollabEval decomposes the evaluation data matrix into dense anchor data and sparse target model samples, reconstructing missing entries with cross-fold matrix completion, enabling rigorous, unbiased estimation via CrossPPI.

Methodology

Matrix Completion for Evaluation

Model evaluation is converted into a collaborative filtering problem: given a partially observed evaluation score matrix $S$, classic low-rank matrix completion (such as iterative SVD or alternative factorizations) is used to impute unobserved entries for the target model(s). This process transposes the problem of expensive prompt-level measurements to one of statistical imputation, relying on empirically observed performance redundancy across models.

Cross-fold completion ensures statistical validity by constructing $K$-fold data splits over the observed prompt labels for the target models, holding out each fold in turn for model training and prediction, analogous to cross-fitting for debiased machine learning estimators.

Cross-Prediction-Powered Inference (CrossPPI)

Using the matrix completion predictions as control variates, CollabEval applies a cross-prediction-powered inference (CrossPPI) framework. The estimator for a target model's mean performance $\theta_i$ is:

$$\hat{\theta}_i = \frac{1}{N} \sum_{j=1}^N \hat{Y}_{ij} + \frac{1}{|\mathcal{J}_i|} \sum_{j \in \mathcal{J}_i} (S_{ij} - \hat{Y}_{ij})$$

where $\hat{Y}_{ij}$ is the completed score and $S_{ij}$ the observed value.

This form guarantees unbiasedness, regardless of the accuracy of the matrix imputation, by correcting prediction bias through the observed entries. Furthermore, the asymptotic variance of $\hat{\theta}_i$ is always less-than-or-equal to standard estimation, provided the imputed values have nonzero correlation with the true scores ("power tuning"), and reduces efficiently as this predictive correlation strengthens.

Empirical Validation

Datasets and Experimental Setup

CollabEval is validated across five representative large-scale evaluation datasets:

• AlpacaEval 2.0 (instruction following)
• MMLU (multiple-choice QA)
• Attributed Question Answering (AQA)
• WMT24++ (multilingual machine translation)
• SWE-bench (code patching)

Dense anchor matrices are built from historical or leave-out system evaluations; multiple protocols (random, paired sampling) and several strong baselines (naive imputation, coreset anchor points, per-prompt means) are considered for comparative analysis.

Statistical Efficiency, Validity, and Coverage

Empirically, CollabEval achieves the following:

• Up to 30% reduction in mean confidence interval (CI) width for pairwise performance differences at low labeled data fractions relative to classical estimators;
• Consistent asymptotic CI coverage (nominal 90%) across datasets, even at low annotation rates;
• Effective sample size increases: to match CollabEval’s estimator variance, the classical method would require up to 40–60% more direct labels;
• Robust performance to anchor set selection and size; gains remain even as anchor pool contracts.

Figure 2: CollabEval attains the smallest CI widths and lowest MSE across benchmarks, both for pointwise and pairwise metrics, while maintaining nominal coverage.

Notably, CollabEval’s efficiency advantage is maximized in high-sparsity regimes (annotation fraction $M$0) and for strongly correlated target-anchor pools. For large $M$1, the benefit diminishes as direct measurements dominate.

Low-Rank Structure of Evaluation Matrices

Explicit singular value analysis shows rapid decay and low effective rank ($M$2 explained >50% variance across tasks), empirically supporting the central premise that inter-model evaluation matrices are amenable to low-rank imputation.

Figure 3: Singular value spectrum analysis of evaluation matrices reveals rapid decay, justifying the use of low-rank matrix completion.

Comparative Ablation and Stability

CollabEval outperforms naive imputation (which introduces bias), untrained per-prompt control variates, and coreset anchor approaches across all metrics. Anchor set selection (top-k, bottom-k, random-k) and matrix completion algorithm (SVD, NN, nuclear norm, IRT) are ablated to show both performance stability and the generality of the approach.

Theoretical Properties

CollabEval provides:

• Unbiasedness under arbitrary matrix completion procedures, as bias correction is always performed;
• Asymptotically valid confidence intervals for both pointwise model means and pairwise comparisons, requiring only a mild algorithmic stability assumption on the completion method;
• Variance Reduction Guarantees: Power tuning ensures that CollabEval’s estimator variance is never worse—and typically strictly better—than standard empirical mean estimators, scaling inversely with the squared correlation between imputed and true values.

These theoretical properties parallel and extend established results for debiased machine learning, multi-center fusion learning, and semi-parametric inference in the presence of missing data.

Practical and Theoretical Implications

Practically, CollabEval enables continuous, large-scale evaluation pipelines to amortize annotation and inference costs by leveraging legacy evaluation data, with immediate relevance for and adoption in LLM benchmarking infrastructure. The method generalizes to arbitrary evaluation metrics and can be layered with further contextual or feature-based completion for richer settings.

Theoretically, this work substantiates a generalized statistical framework for collaborative evaluation: moving beyond the limitations of item response theory and coreset selection, it unifies low-rank matrix modeling, control-variate debiasing, and prediction-powered semi-parametric inference. The approach is robust to misspecification, enables compositional estimands, and matches the efficiency of more restrictive methods.

Future Directions

Key future directions include:

• Extension to contextualized and prompt-conditioned settings using covariate-augmented matrix/tensor completion;
• Online updating of the anchor set as new evaluations and model classes emerge;
• Joint modeling of evaluation item (prompt) difficulty and model ability;
• Deployment for dynamic, “always-on” benchmarks and leaderboard monitoring;
• Theoretical exploration of minimax efficiency and simultaneous confidence regions over benchmarked model families.

Conclusion

CollabEval delivers a statistically principled and empirically validated methodology for collaborative model evaluation, formalizing and exploiting redundancy across historical model outputs. Substantial practical annotation and inference savings are achieved without sacrificing statistical rigor, making CollabEval highly relevant for efficient, robust benchmarking of rapidly developing generative models. The framework establishes a general foundation for statistically-efficient evaluation under data sparsity, supporting both immediate adoption and future theoretical generalization.