Submodular Benchmark Selection

Abstract: Evaluating LLMs across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small, informative subset as submodular maximization under a multivariate Gaussian model. Entropy (log-determinant covariance) and mutual information between selected and remaining benchmarks arise as natural objectives. Both are submodular; entropy selection coincides with pivoted Cholesky and has spectral residual bounds, while mutual information is non-monotone in general but empirically monotone for small subsets, so we optimize it greedily. Experiments on three matrices from ten public leaderboards show that mutual information selection outperforms entropy for imputation at small subsets.

• The paper formulates benchmark selection as submodular maximization under a multivariate Gaussian model, using entropy and mutual information to choose informative evaluation subsets.
• Mutual-information selection delivers stronger small-budget predictions: on MMLU, five of 57 benchmarks achieve an imputation R² of 0.91, while one reaches 0.83.
• The results show that dense benchmark matrices are highly redundant, but sparse or heavily missing data can undermine covariance-based selection, making uncertainty handling and robust alternatives important.

Motivation and problem statement

Evaluating LLMs across the full landscape of public benchmarks is expensive, and much of that expense is redundant because benchmark scores are strongly correlated. This paper formalizes the question of which small subset of benchmarks to actually run as submodular maximization under a multivariate Gaussian model of the score matrix. Given $M$ models evaluated on $N$ benchmarks, each model's score vector is modeled as an independent draw from $\mathcal{N}(\mu, \Sigma)$; selecting a subset $A$ then amounts to choosing which coordinates to observe, with unobserved scores imputed via Gaussian conditional expectations (2605.02209).

Two objectives are considered, both inherited from the Gaussian sensor placement framework of Guestrin, Krause, and Singh: the entropy $H(X_A) = \tfrac{1}{2}\log\det(2\pi e\,\Sigma_{AA})$ and the mutual information $I(X_A; X_{\bar A})$. Both are submodular. The key structural difference from sensor placement is that here the covariance is estimated from data — the score matrix provides many independent draws — rather than specified a priori through a kernel.

Algorithms

Entropy selection is pivoted Cholesky. Greedy entropy maximization selects at each step the benchmark with maximum residual conditional variance, maintained via incremental Cholesky updates in $O(k^2 N)$ total time. The paper proves (Lemma 1) that this algorithm is exactly pivoted Cholesky, that its residual variance equals the pivoted-Cholesky residual trace, and that under a modular shift $H_c(X_S) = H(X_S) + c|S|$ making marginals non-negative, it inherits the standard $(1-1/e)$ monotone-submodular guarantee without changing greedy choices.

Mutual information requires complement refactorization. The MI marginal gain involves two conditional variances: one for the growing selected set (handled by forward Cholesky updates) and one for the shrinking complement, obtained from the precision diagonal $P_{vv} = [(\Sigma_{\bar A,\bar A})^{-1}]_{vv}$. Rank-one precision downdates exist but are numerically unstable when $N$0 is small; the paper instead performs a fresh Cholesky factorization of the complement block per step, costing $N$1 overall — negligible for hundreds of benchmarks (under one second on all datasets). MI is non-monotone in general, so no approximation guarantee applies; the authors use greedy as a heuristic, justified empirically by positive marginal gains at all steps for small $N$2.

Missing data. Incomplete score matrices are handled with EM for Gaussian missing data under a MAR assumption, including PSD projection, Ledoit–Wolf shrinkage for rank-deficient regimes ($N$3), and an imputation-uncertainty correction in the M-step. Convergence takes 2–3 iterations on dense MMLU but roughly 300 iterations on sparse matrices.

A spectral diagnostic connects subset size to eigenvalue decay: since no coordinate subset can beat the best rank-$N$4 approximation, the eigenvalue tail lower-bounds any selection's residual variance, and fast decay indicates that small subsets should suffice.

Experiments

Three score matrices are assembled from ten public leaderboards: MMLU ($N$5, fully observed), MTEB ($N$6, 77% observed), and Merged ($N$7, 31% observed), plus the BenchPress matrix ($N$8, 33.8% observed) in an appendix. Evaluation uses 10-fold cross-validation over models, with imputation $N$9 computed in standardized space.

The eigenspectrum analysis shows benchmark scores live in a very low-dimensional subspace: two components capture 90% of variance across 57 MMLU subjects, six suffice for MTEB, and eight for Merged.

Imputation results are strong where data is dense:

Dataset	$\mathcal{N}(\mu, \Sigma)$0 at $\mathcal{N}(\mu, \Sigma)$1	$\mathcal{N}(\mu, \Sigma)$2 at $\mathcal{N}(\mu, \Sigma)$3
MMLU	0.89–0.91	0.95
MTEB	~0.72–0.76	~0.85
Merged	~0.35–0.53	~0.55–0.59

On MMLU, performance is stable across holdout fractions: even training on only ~545 models yields $\mathcal{N}(\mu, \Sigma)$4 at $\mathcal{N}(\mu, \Sigma)$5. With MI selection, five benchmarks out of 57 achieve $\mathcal{N}(\mu, \Sigma)$6, and a single benchmark reaches $\mathcal{N}(\mu, \Sigma)$7.

Entropy versus mutual information. MI outperforms entropy at small $\mathcal{N}(\mu, \Sigma)$8 on all three matrices — the gap is largest at $\mathcal{N}(\mu, \Sigma)$9 on MMLU ($A$0 vs. $A$1) and roughly 10 $A$2 points on MTEB for $A$3 — while entropy overtakes for larger $A$4 on well-conditioned data (crossover near $A$5 on MMLU). The paper identifies a "surrogate gap": entropy achieves lower residual variance than MI yet worse imputation at small $A$6, because entropy rewards diversity among selected benchmarks whereas MI rewards coupling with the unselected complement, which is what drives prediction quality. Selection-order analysis reinforces this: entropy picks high-variance "outlier" subjects (abstract algebra, college mathematics), while MI picks "hub" subjects (miscellaneous, professional psychology) with strong predictive links to the rest. Selection is highly stable on MMLU (first nine MI picks identical across all folds) but unstable on Merged, reflecting noisy covariance estimation under 69% missingness.

Random baselines and negative results. Random selection is surprisingly competitive in $A$7, especially on highly redundant MMLU, though it is dominated in residual variance and achieved MI. Two appendix experiments strengthen the case for the linear Gaussian pipeline: zero-shot TabImpute imputation substantially degrades $A$8 (e.g., $A$9 vs. $H(X_A) = \tfrac{1}{2}\log\det(2\pi e\,\Sigma_{AA})$0 on MMLU with MI at $H(X_A) = \tfrac{1}{2}\log\det(2\pi e\,\Sigma_{AA})$1), and a logit-space transform fails to improve results anywhere, slightly reducing them on MTEB, Merged, and BenchPress. On the sparse BenchPress matrix, both principled methods fail — random selection beats unstable MI — confirming that the rank-deficient, heavily missing regime limits what covariance-based selection can extract.

Limitations and open questions

Several caveats bear directly on the headline numbers. Formal normality tests reject the Gaussian assumption on essentially every benchmark (all 57 MMLU and 56 MTEB benchmarks reject under Shapiro–Wilk with BH correction); the authors argue the conditional mean remains the best linear predictor regardless, so the Gaussian model is a working approximation rather than a distributional claim, but robust alternatives such as copulas remain unexplored. The MAR assumption for leaderboard missingness is acknowledged as approximate, since missingness is often model- and benchmark-dependent. The MI guarantee gap is real: greedy MI has no approximation bound, and its empirical monotonicity holds only for small $H(X_A) = \tfrac{1}{2}\log\det(2\pi e\,\Sigma_{AA})$2. Sparse matrices require shrinkage that dampens cross-benchmark correlations, and the BenchPress results show the pipeline can fail outright in that regime. Finally, the evaluation conditions only on benchmarks already observed for validation models, testing prediction from existing coverage rather than simulating deployment where all selected benchmarks would be newly run. Open questions include the optimal granularity of benchmarks (few large vs. many small) and robustness to contamination from fine-tuning on benchmark-adjacent data.

Conclusion

The paper recasts benchmark selection as submodular maximization, showing that entropy selection coincides exactly with pivoted Cholesky and carries shifted monotone guarantees, while MI — optimized greedily without guarantees — delivers better imputation at small budgets by targeting coupling with the unselected complement. Empirically, a handful of well-chosen benchmarks captures most evaluative signal on dense score matrices, with diminishing returns in sparse, heterogeneous settings. The practical recommendation is MI selection for budget-constrained evaluation, entropy for diversity or residual-variance reduction, with plain Gaussian conditional-mean imputation outperforming more elaborate nonlinear alternatives.