LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

Abstract: LLM evaluation platforms increasingly rely on pairwise human judgments. These data are noisy, sparse, and non-uniform, yet leaderboards are reported with limited uncertainty quantification. We study this as semiparametric inference for a low-rank latent score tensor observed through pairwise comparisons under Bradley-Terry-Luce-type models. This places LLM evaluation in a new tensor completion setting with structured observations, non-uniform sampling, and pairwise contrasts. Our target is a smooth functional $ψ(T^\star)$, including linear estimands such as ability gaps and nonlinear ones such as win probabilities. We derive the information operator on the low-rank tangent space, the efficient influence function, and the semiparametric efficiency bound, then construct a one-step debiased estimator with asymptotic normality. A central challenge is that the information operator is anisotropic and does not commute with the tangent-space projection, creating a bottleneck absent from isotropic models. We introduce a score-whitening method that equalizes local Fisher information and restores stable inference at the optimal sample-complexity scale. Our results provide a principled framework for uncertainty quantification in LLM evaluation and more broadly for inference on low-rank structures from pairwise data.

• The paper introduces a semiparametric efficiency framework that models LLM evaluation as a low-rank tensor completion problem using pairwise human comparisons.
• It develops a cross-fitted, score whitening one-step estimator to tackle challenges arising from non-uniform sampling and anisotropic information geometry.
• Empirical validations demonstrate that the proposed method yields tighter confidence intervals and improved inference compared to traditional IPW approaches.

Semiparametric Efficiency for LLM Evaluation via Low-Rank Tensor Completion

Introduction and Problem Setting

The paper "LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency" (2604.05460) investigates the statistical foundations of LLM evaluation based on pairwise human comparisons, a paradigm exemplified by platforms such as Chatbot Arena. The evaluation process is formalized as inference on a latent low-rank score tensor, with the data comprising noisy, sparsely and non-uniformly sampled pairwise preference comparisons embedded in intricate, contextual dimensions (models, tasks, users, languages, etc). Existing leaderboards aggregate these outcomes for ranking but offer inadequate uncertainty quantification and do not leverage the structured regularity inherent in LLM performance across multiple axes.

The central research objective is statistically efficient inference—primarily, estimation and valid confidence intervals for smooth (linear and nonlinear) functionals of the latent performance tensor $T^\star$, exploiting low-rank structure, under non-uniform and context-dependent observation patterns with noisy binary outcomes.

Figure 1: LLM Arena's human evaluation pipeline; left: pairwise comparison interface, right: resulting leaderboard with substantial sampling imbalance.

Theoretical Framework: Low-Rank Tensor Model and Pairwise Observation

Formally, given $d_1$ models, $d_2$ task categories, etc., the unknown $T^\star$ is a tensor in $\mathbb{R}^{d_1 \times \cdots \times d_m}$ admitting a low-rank Tucker decomposition. Each observation yields a label indicating which of two models is preferred in a specified context, modeled by a generalized Bradley-Terry-Luce (BTL) contrast:

$$P(Y=1|X; T^\star) = \sigma(\langle T^\star, X\rangle)$$

where $X$ encodes the comparison, and $\sigma$ is the logistic function. Identifiability restrictions (e.g., sum-to-zero constraints along the model axis) are imposed since only differences of tensor entries are observed.

Inferential targets are smooth functionals $\psi(T^\star)$, including:

• Linear contrasts: Ability gaps, category-specific means ($\psi(T^\star) = \langle \Gamma, T^\star \rangle$)
• Nonlinear functionals: Win probabilities (e.g., $d_1$0), calibrated/aggregated scores

Semiparametric Efficiency, Information Geometry, and the Non-commutativity Bottleneck

The core technical contribution is a semiparametric efficiency theory for inference under the pairwise observation model:

• Tangent space: Statistical inference can only occur along directions compatible with the low-rank and identifiability constraints.
• Information operator: Fisher information is aggregated over the random sample of observed pairs and is highly anisotropic due to the heterogeneity in both design and local score gaps.
• Efficiency bound: For a linear functional, the asymptotic variance is

$d_1$1

representing the semiparametric analog of the Cramér-Rao lower bound.

• Efficient Influence Function (EIF): The EIF is characterized via operator equations in the tangent space, capturing first-order sensitivity to local perturbations (cf. efficient score functions).

A central technical challenge arises because, under non-uniform sampling and heterogeneous matchups, the information operator $d_1$2 does not commute with the tangent projector $d_1$3, so $d_1$4 can exhibit substantial off-diagonal and entrywise amplification, elevating sample complexity for efficient estimation. This phenomenon is absent in isotropic models with additive or homoscedastic noise.

One-Step Estimation, Score Whitening, and the Optimal Trade-off

The proposed estimator is a cross-fitted, one-step correction to a plug-in low-rank estimator, leveraging the structure of EIFs. The analysis precisely delineates the sample complexity bottlenecks due to the non-commutativity of $d_1$5 and addresses them with a score whitening approach, which normalizes each comparison by its local Fisher information:

• The whitened score essentially transforms the Fisher-weighted problem into an isotropic regime, enabling optimal inference rates and eliminating dimension-dependent amplification.
• For arbitrary non-uniform sampling, an inverse-probability weighting (IPW) scheme is developed, restoring uniformity to the Gram structure.
• The framework is extended from linear to nonlinear smooth targets via local Taylor linearization and plug-in EIF gradients.

  Figure 2: Entrywise refinement after alternating minimization delivers a sharp reduction in Frobenius and $d_1$6 error in the initial estimator, a critical prerequisite for efficient one-step correction.

Empirical Validation: Coverage, Calibration, and Practical Implications

Extensive simulations and real-data experiments validate the theoretical predictions:

• The efficient one-step and score-whitened estimators achieve high-fidelity coverage for confidence intervals on both linear and nonlinear targets under varying sample sizes, low-rank structure, outcome heteroskedasticity, and non-uniform observation (see Figure 3 and Figure 4).
• The efficiency gain—i.e., the narrowing of confidence intervals—of the efficient estimator over the IPW/score-whitened alternatives becomes more pronounced with increasing sample size and lower observation noise anisotropy. In realistic LLM leaderboard data, where comparisons are highly imbalanced, the full semiparametric procedure provides the tightest inference.
• Empirically, entrywise refinement (see Figure 2) is essential: naively fitting independent BTL models to each context induces severe error inflation in sparse regimes, while low-rank structure enables sharing of statistical strength and effective dimension reduction.

  Figure 3: Coverage and variance calibration for the entry statistic as a function of $d_1$7, demonstrating near-nominal confidence intervals and accurate plug-in variance estimation under increasing sample size.

  

  Figure 4: Coverage and variance calibration for entry targets as a function of rank $d_1$8, showing robustness of inference procedures across model complexity.

Broader Theoretical and Practical Implications

• Link to Matrix/Tensor Completion: The EIF calculation and information geometry directly generalize the classical low-rank matrix completion results, but here under general non-Gaussian, 1-bit, or pairwise noise---thus connecting LLM leaderboard analysis to a broader statistical literature that includes collaborative filtering, multitask estimation, and indirect measurement models.
• Statistical Foundations for Public Leaderboards: The presented methodology justifies uncertainty quantification on LLM leaderboards even under highly incomplete, non-uniform, and noisy crowdsourced feedback, yielding statistically valid confidence intervals for any structured performance contrast or nonlinear summary.
• Guidance for Platform Design: The inefficiency induced by sampling imbalance and non-uniform informative matchups quantifies how design choices or post-hoc reweighting affect CI width. Score whitening and IPW correction are recommended in practice when full knowledge of the information geometry is unattainable or estimation of $d_1$9 is unstable.
• Future Directions: The results highlight further open directions: optimal experiment (comparison) design in the face of cost or ethical constraints; robustification to heavy-tailed user or prompt distributions; adaptation to rank or structural model misspecification; and extension to multiway or non-binary feedback.

Conclusion

This work provides a mathematically rigorous, practically relevant framework for uncertainty quantification and efficient inference in LLM evaluation via human pairwise comparisons. By connecting leaderboard estimation to the semiparametric statistics of low-rank tensor completion with indirect observation, it resolves the technical intricacies of structured, sparse, and non-isotropic information geometry. The framework yields clear guidelines for estimator construction, confidence interval computation, and leaderboard interpretation, with direct implications for empirical AI evaluation, benchmarking, and continuous model development.