You Don't Need to Run Every Eval

Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release. But do we need to run every eval? We compile a public score matrix of 84 frontier models on 133 benchmarks (2,604 cells, 23.3% filled) and find it is approximately rank-2: a model's scores across all 133 benchmarks are largely determined by just two numbers. We confirm this in two ways: scores hidden from the matrix are best recovered using two factors, and two factors already explain over 90% of the variation among models on the benchmarks they share. Building on this, we design BenchPress: a logit-space rank-2 matrix completion method that recovers held-out scores to within 4.6 points, and a confidence layer that says when each prediction can be trusted. Using BenchPress, we find a subset of five benchmarks {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} that can recover the rest of a model's public scorecard to within 3.93 points. For a tighter inference budget, a cheaper set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} can predict a model's evals to within 4.55. We release the score matrix, the BenchPress code, and an interactive tool that predicts any model's score on any benchmark.

• The paper demonstrates that a rank-2 latent structure explains over 90% of LLM performance variance across diverse benchmarks.
• The authors introduce BENCHPRESS, a logit-space ALS matrix completion algorithm that infers missing scores with a median error of ~4.6 points.
• Using a small, strategically chosen probe set, the approach preserves model rankings and enables cost-effective, reliable LLM evaluation.

Summary of "You Don't Need to Run Every Eval" (2606.24020)

Motivation and Problem Formulation

The costly and redundant nature of LLM benchmark evaluation has become evident as modern frontier model releases report scores across dozens of benchmarks. Each evaluation run incurs significant monetary and wall-clock costs, especially considering the repetition needed for model selection, training progress tracking, and deployment decisions. The paper investigates whether it's necessary to conduct every individual benchmark evaluation or if a smaller subset can efficiently infer the remainder of a model's evaluation suite with high fidelity.

Score Matrix Construction and Low-Rank Structure

A comprehensive public score matrix is compiled containing 84 recent frontier models (spanning 13 providers) evaluated on 133 benchmarks. The matrix is sparse (23.3% filled) but encompasses a broad benchmark spectrum: math, coding, agentic, multimodal, factuality, preference, and safety, among others.

Critical analysis reveals that this matrix is effectively rank-2, i.e., two latent factors have the capacity to explain over 90% of performance variance across models and benchmarks. This finding is supported by two lines of evidence:

• Soft-Impute rank sweep: Median absolute percentage error (MedAPE) on held-out data is minimized at rank 2 both for raw-score and logit-transformed spaces.
• SVD: Singular value decompositions of fully-observed submatrices consistently show that two components dominate variance.

The operational implication is that a model's scores across disparate benchmarks are essentially determined by two numbers, refuting the independence assumption of benchmark scores.

BENCHPRESS: Logit-Space Low-Rank Matrix Completion

Building on the observed low-rank structure, the authors develop BENCHPRESS, a logit-transformed, bias-decomposed alternating least squares (ALS) matrix completion algorithm. Its recipe:

1. Apply logit transform to percentage scores, standardize each column.
2. Fit a rank-2 bias-decomposed ALS, using a global offset, row (model) offset, column (benchmark) offset, and a rank-2 residual correction.
3. Invert the standardization and the logit transform to map predicted scores back to original scale.

BENCHPRESS achieves a median absolute error (MedAE) of 4.63 points on held-out entries at full coverage. Regression-based alternatives provide marginally better MedAE but at lower coverage, while BENCHPRESS offers deterministic, full-coverage prediction.

Benchmark Probe Selection and Scorecard Recovery

A key practical application is budgeted scorecard recovery: selecting a small probe set of benchmarks to run, and using BENCHPRESS to infer the rest. Using a greedy selection strategy, five benchmarks (GPQA Diamond, HLE, Codeforces, MMLU-Pro, ARC-AGI-1) recover the remainder of a model’s scorecard to within 3.93 points MedAE (or 4.55 with low-cost probes), outperforming random probe sets by significant margins.

Notably, reasoning and math-oriented benchmarks dominate probe sets, reflecting their high mutual predictivity and alignment with the principal axes of matrix variance.

Ranking Preservation

The preservation of pairwise model rankings is evaluated. With a five-point margin on true scores, BENCHPRESS-completed scores match the actual orderings 92.1% of the time for same-benchmark model pairs. For practical deployment, this validates the use of matrix-completed scores for comparative evaluation and candidate shortlisting under uncertainty tolerances.

Temporal Generalization for Deployment

For newly released models absent from the training matrix, BENCHPRESS can still predict scorecards reliably after a small seed evaluation. With five revealed scores, MedAE drops to 4.83 points; with ten, it falls to 2.57. The error distribution narrows as more probes are observed, demonstrating robust temporal transfer given sufficient anchor points.

Prediction Reliability and Trustworthiness

Prediction reliability is characterized by both benchmark-side and model-side factors:

• Benchmark-side: Prediction becomes harder with wider score spread (across models), fewer observed model scores, and absence of strongly correlated benchmark neighbors.
• Model-side: Reasoning models and higher-scoring models are easier to predict, as are models with many observed scores and correlated peers, and those anchored by recent models in the training matrix.

A hybrid reliability estimator (using both ensemble spread and matrix support features) reliably identifies low-risk predictions, achieving selective MedAE as low as 1.83 points for the safest 20% of predictions. This estimator enables practitioners to triage which predictions are trustworthy enough to substitute for benchmark runs.

Limitations and Future Directions

The approach is inherently snapshot-dependent: expansion of the matrix or emergence of models with new capability profiles may alter the rank-2 geometry underlying BENCHPRESS. Prediction is only as good as benchmark construction and reporting; for noisy or poorly specified benchmarks, predictions faithfully reproduce noise. Application to instance-level benchmark outcomes or specialized domains (audio, robotics, scientific simulation) remains unexplored. Integrating metadata—model architecture, training data, external features—could further anchor predictions for outliers. Regular recalibration of probe sets and rank selection is required as the matrix evolves.

Conclusion

The paper demonstrates that large-scale LLM benchmark evaluation is highly redundant and governed by a low-dimensional latent structure. BENCHPRESS provides principled, efficient matrix completion based on logit-space ALS, enabling substantial reduction in evaluation effort, robust scorecard recovery, and preservation of model ranking decisions with strong numerical guarantees. The practical implications are significant: practitioners can conduct evaluations with a small subset of informative benchmarks and rely on matrix completion for the remainder, subject to explicit reliability characterization. Future work should address cross-snapshot stability, extension to new domains, integration of model metadata for outlier prediction, and instance-level evaluation compression.