Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization

Abstract: Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models' strengths and weaknesses. Such fine-grained evaluations, however, suffer from a data bottleneck: human gold-standard labels are too costly at this scale, while automated ratings are often misaligned with human judgment. To resolve this challenge, we propose a novel statistical model based on tensor factorization that merges cheap autorater data with a limited set of human gold-standard labels. Specifically, our approach uses autorater scores to pretrain latent representations of prompts and generative models, and then aligns those pretrained representations to human preferences using a small calibration set. This sample-efficient methodology is robust to autorater quality, more accurately predicts human preferences on a per-prompt basis than standard baselines, and provides tight confidence intervals for key statistical parameters of interest. We also showcase the practical utility of our method by constructing granular leaderboards based on prompt qualities and by estimating model performance solely from autorater scores, eliminating the need for additional human annotations.

• The paper introduces a tensor factorization model that efficiently aligns autorater signals with minimal human calibration to assess generative model performance.
• It leverages low-rank decompositions and ordered logit regressions to derive prompt-specific insights and reliable uncertainty estimates.
• Empirical results demonstrate improved performance on benchmarks with only 10% human annotation, enabling robust leaderboards and fine-grained model comparisons.

Efficient Fine-Grained AI Evaluation via Tensor Factorization

Introduction and Motivation

This paper, "Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization" (2603.02029), addresses the critical challenge in generative model evaluation: acquiring robust, fine-grained insights despite the prohibitive cost of prompt-level human annotation. Traditional evaluation methodology aggregates performance across heterogeneous prompts, obfuscating subtle model weaknesses and strengths. Fine-grained evaluations, informed by psychometric paradigms such as Item Response Theory (IRT), enable the identification of multidimensional skill profiles and facilitate dynamic applications, such as model routing based on prompt difficulty. However, their practical deployment is hindered by the sheer scale of required human annotation and the unreliability of current automated rating systems ("autoraters") at the prompt level.

The authors propose a statistically grounded tensor factorization approach that efficiently aligns cheap autorater signals with sparse, high-quality human labels to facilitate prompt-level characterization of generative models. This framework leverages autorater scores to pretrain latent representations for models and prompts, calibrating them to human preferences using a minimal calibration set. The methodology yields not only accurate per-prompt human-aligned predictions but also reliable confidence intervals for statistical inference, offering robust leaderboards at varying degrees of granularity.

Methodological Framework

Tensor Factorization Model

The central object is a capability tensor $\Psi \in \mathbb{R}^{I \times J \times K}$, where $\Psi_{i,j,k}$ quantifies the performance of model $i$ on prompt $j$ as perceived by rater $k$. The model assumes low-rank structure in $\Psi$ via CP decomposition: prompt-model-rater interactions are explained as linear combinations of latent skills, with $\Theta \in \mathbb{R}^{I\times R}$ (model skill matrix), $A \in \mathbb{R}^{J\times R}$ (prompt demand matrix), and $\Gamma \in \mathbb{R}^{K\times R}$ (rater criterion matrix), so that $$\Psi_{i,j,k} = \sum_{r=1}^R \Theta_{i,r} A_{j,r} \Gamma_{k,r}$$.

Ordinal evaluation outcomes are modeled via ordered logit regressions, with rater-specific cutoffs. Pairwise evaluation (side-by-side) and pointwise templates are both supported, utilizing appropriate transformations of $\Psi$. Model fitting proceeds in two stages: 1) representation learning on abundant autorater data; 2) calibration of rater parameters via human labels. Optional fine-tuning on human data further improves empirical accuracy, at the expense of standard interval validity.

Empirical Evaluation

Predictive Power

The proposed method is benchmarked against three evaluation suites: Gecko (text-to-image), BigGen Bench (text-generation), and LMArena (side-by-side LLM evaluation). Autorater data is collected using template variation and persona diversification, including both single-sided and pairwise templates. The methodology achieves lower cross-entropy losses across human budgets relative to baselines (constant, prompt-specific, Prompt-to-Leaderboard), demonstrating improved fit with limited human annotation.

Figure 1: Test cross-entropy loss comparison, showing consistent advantage of tensor factorization's prompt-specific modeling and efficient use of autorater signals.

Fine-Grained and Category-Specific Insights

Model rankings are extracted for highly cohesive prompt categories using only 10% of available human data. Results yield significant separation of model performance profiles within these categories—e.g., Imagen matches SDXL on compositional tasks but underperforms in additive object prompts; LLaMa-2-13b and GPT-3.5-Turbo exhibit distinct strengths in instruction following versus reasoning.

Figure 2: Category cohesion rankings for Gecko and BGB, illustrating quantitative cohesion and enabling robust subgroup evaluation.

Prompt-Level Model Strengths and Weaknesses

Fine-grained comparisons across prompts further illuminate task-dependent model strengths. For Gecko, Imagen excels in text rendering where Muse is comparatively weak, while Muse dominates object counting tasks. In BigGen Bench, GPT-3.5-Turbo outperforms LLaMa-2-13b on reasoning but is matched on safety/instruction. LMArena experiments show LLaMa-3.3-70b ties Gemini-2.5-Pro in ~24% of prompts and beats it outright in ~8%, revealing non-obvious overlap in model capability.

Figure 3: Fine-grained comparison of model capabilities with 10% human annotation, highlighting Imagen and Muse's trade-offs in Gecko, and GPT-3.5-Turbo's reasoning advantage in BGB.

Figure 4: LMArena estimates demonstrate LLaMa-3.3-70b's capacity to match or outperform Gemini-2.5-Pro in a non-trivial fraction of prompts.

Autorater-Only Model Performance Prediction

Leave-one-out experiments show that average model scores and win-rate differences can be precisely predicted for models with no observed human annotations, solely leveraging autorater signals fused via tensor factorization. This validates the model's transfer capability and justifies its application in cold-start scenarios.

Figure 5: Predictive validation on held-out models across three benchmarks, demonstrating preservation of pairwise ranking and strong correlation with ground truth.

Interpretability and Benchmark Analysis

Prompt embeddings derived from the factorization framework enable interpretable insights into benchmark composition. For example, analysis of prompt length versus model rank in Gecko illustrates SD1.5's preference for short prompts, highlighting limitations in model generalization and guiding future prompt selection strategies.

Theoretical and Practical Implications

The tensor factorization approach provides statistically principled, sample-efficient calibration of human-aligned model evaluation under severe annotation scarcity. Practically, it enables construction of granular leaderboards and the identification of fine-grained tradeoffs critical in deployment and model routing. Theoretically, the method supports uncertainty quantification via approximate confidence intervals and simultaneous coverage for leaderboards, although limitations exist regarding identifiability and error propagation after optional fine-tuning.

The framework suggests future directions, including active learning for optimal calibration, leveraging latent capability estimates as reward signals for RLHF, and extending to evaluation of autonomous agents and complex modalities (video, code) where prompt-response pairs are insufficient for capability assessment.

Conclusion

This work delivers a robust methodology for prompt-level, human-aligned evaluation of generative models, efficiently merging cheap signals from autoraters with minimal human calibration via tensor factorization. The empirical results establish significant improvement in predictive accuracy and uncertainty quantification compared to traditional baselines, especially in data-scarce regimes. The approach enables actionable, fine-grained leaderboards and interpretable model insights, supporting both theoretical research and practical deployment. Several directions remain open, particularly regarding adaptive calibration, reward shaping for RLHF, and generalization to agentic workflows in multimodal settings.