Efficient Benchmarking Is Just Feature Selection and Multiple Regression

Abstract: Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman $ρ$ and Kendall $τ$) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits. Tutorial code can be found at https://github.com/sambowyer/mrmr_eval .

• The paper presents efficient LLM benchmarking as a regression and feature selection challenge using mRMR and kernel ridge regression.
• It empirically demonstrates that mRMR++ achieves lower RMSE, higher ranking correlation, and significant compute savings compared to traditional methods.
• The approach offers practical and theoretical benefits, enabling robust, scalable LLM evaluation with minimal coresets and improved representativity.

Recasting Efficient LLM Benchmarking as Feature Selection and Multiple Regression

Problem Setting and Motivation

The exponential growth in LLM size and complexity has rendered standard evaluation procedures computationally infeasible, especially when assessing models on large-scale benchmarks involving hundreds to thousands of examples, or for settings requiring repeated ablations and checkpoint evaluations. The primary challenge is to efficiently estimate a model's performance on a full benchmark using only a subset ("coreset") of the benchmark's questions—for both practical (compute savings) and theoretical (representativeness, generalizability) reasons.

This paper conceptualizes efficient benchmarking as a classic statistical problem: regression with feature selection. Formally, the goal is to (1) select a subset of benchmark questions (feature selection), and (2) learn a predictive function mapping responses on the coreset to the full benchmark score (multiple regression). The authors argue—and demonstrate empirically—that principled application of feature selection via minimum redundancy maximum relevance (mRMR) and kernel ridge regression (KRR) yields superior results compared to the state-of-the-art, including methods based on clustering, item response theory (IRT), sampling, and adaptive selection.

Methodology: mRMR Feature Selection and Kernel Ridge Regression

Feature Selection via mRMR

Feature selection is performed using the mRMR algorithm, which optimizes the mutual information quotient (MIQ): select features/questions that maximize their mutual information with the target (full benchmark score, denoted $Y$), while minimizing redundancy among selected features. The MIQ objective is greedy, iteratively extending the coreset by the question whose relevance to $Y$ divided by average redundancy to the current coreset is maximal.

The paper implements both discrete-discrete and discrete-continuous MI estimation (utilizing Kraskov–Stögbauer–Grassberger, PCA-corrected estimators, and Ross's algorithm), with empirical tuning of the nearest-neighbors parameter $k$. Binary-performances use empirical estimation over model responses; continuous metrics are handled via adapted estimators.

Prediction via Ridge and Kernel Ridge Regression

Regression is performed using classical ridge regression and kernel ridge regression (KRR) with polynomial kernels up to degree $d$, enabling the model to capture not only additive effects of individual questions, but also synergistic multiplicative effects (e.g., $d=2$ enables interaction terms between pairs of questions).

Prediction is performed on new/test models using the learned mapping from coreset scores to the mean or summary benchmark score. Cross-validation is used for regularization parameter selection.

Comparative Evaluation and Results

The method is evaluated across multiple datasets:

• Binary Metrics: Open-LLM-Leaderboard and HELM-Lite, comprising hundreds of models evaluated on benchmarks like IFEval, OpenLLM-Math, MMLU-Pro, ARC-Challenge, BBH, GPQA, MuSR, CommonsenseQA, GSM8K, LegalBench, MATH, Med-QA, MMLU.
• Continuous Metrics: Summarization datasets (GovReport, BioLaySumm), with metrics such as ROUGE-L, BERTScore, and FKGL.
• Pass@$k$ Metrics: Python coding benchmarks (MBPP, MBPP+, LBPP, HumanEval) with varying $k$.

Strong baselines include random sampling, exhaustive random search, K-means clustering-based approaches (AnchorPoints), IRT-driven methods (gp-IRT, Metabench), Lasso regression, and variants using alternative regression methods.

Strong Numerical Claims

• mRMR/KRR Outperforms Competitors: Across nearly all settings (binary, continuous, pass@$k$), mRMR++ (MIQ variant with KRR, $d=2$) achieves lower RMSE and MAE, as well as higher ranking correlations (Spearman $\rho$, Kendall $Y$0), particularly with moderate source-model pool sizes ($Y$1).
• Enhanced Ranking Correlation: Even with small coresets (e.g., 5% of the benchmark), ranking correlations remain high—indicating robust model ordering and comparability.
• Speed and Stability: mRMR-based coreset construction is orders-of-magnitude faster than clustering or IRT inference, with greater stability (measured via $Y$2 and Hamming metrics)—the coreset shows high overlap across seeds and data splits.
• KRR Improves Other Methods: Kernel ridge regression, when applied to AnchorPoints/TinyBenchmarks-generated coresets, improves their predictive performance—even outperforming their original aggregation strategies.
• Quadratic Kernels Optimal: Degree $Y$3 polynomial kernels strike a balance between capturing pairwise effects and avoiding overfitting, outperforming higher-degree variants.
• Coreset Difficulty: mRMR selects questions of intermediate difficulty, focusing on maximal information content rather than broad representativity.

  Figure 1: mRMR++ consistently delivers lower RMSE and higher Kendall $Y$4 compared to baselines, for both binary and pass@$Y$5 benchmarks.

  

  Figure 2: For $Y$6, mRMR++ dominates across coreset sizes in prediction error and ranking correlation; performance degrades for very small $Y$7.

  

  Figure 3: mRMR selects questions with a balanced spectrum of difficulty compared to competitor methods; the stability and timing advantages are notable.

  

  Figure 4: True vs. predicted summary scores on ARC-C, showing that kernel ridge regression yields low training RMSE and reduced prediction variance.

Analytical Insights and Ablations

Extensive ablation experiments cover:

• MI Estimator Parameters: Optimal $Y$8 for binary, $Y$9 for continuous; MIQ variant outperforms MID, and F-statistic-based selectors underperform.
• Regression Method Variants: KRR ($k$0) outperforms alternatives (Lasso, Random Forests, logit transformation).
• Relevance Target Options: Using summary score as the target is optimal; first principal component alternatives perform worse.
• Underspecification Effects: Performance gains from polynomial ridge regression diminish as $k$1 decreases—underspecified settings reduce signal-to-noise.
• IRT Dimensionality: For binary, $k$2 is generally best; for continuous, results are less sensitive, but higher-dimensional continuous beta-IRT shows instability.

  Figure 5: Optimality of quadratic kernels ($k$3) for all methods; higher degrees degrade prediction.

  

  Figure 6: mRMR scheme and MI estimator hyperparameter $k$4 ablations: MIQ and PCA-KSG estimators yield best performance.

  

  Figure 7: mRMR with MIQ achieves superior relevance-redundancy ratios for coreset selection.

Practical and Theoretical Implications

The approach provides substantial compute savings, enabling rigorous model evaluation at scale for in-distribution settings such as training checkpoint tracking or inference ablation studies. The focus on static coresets is motivated by practical needs—batch processing, interpretability, and throughput. By reframing efficient benchmarking through the lens of classic feature selection and regression, well-developed statistical tools from supervised learning theory become applicable.

From a theoretical standpoint, these results challenge the necessity—and complexity—of more elaborate clustering or IRT-based approaches for the majority of real-world LLM evaluation workloads. The demonstrated generalization and stability of mRMR/KRR-based coresets, even to related metrics (pass@$k$5 for varying $k$6), indicate powerful representational properties of the selected coreset.

Figure 8: Performance benefits from polynomial ridge regression are sensitive to underspecification ratios $k$7.

Future Directions

Unsolved issues and future avenues include:

• Continuous Metrics: mRMR’s performance on continuous datasets remains sub-optimal; enhancements may be feasible via refined MI estimators or alternative feature importance functions.
• Out-of-Distribution and Adaptive Settings: Extending the approach to OOD scenarios (test models outside training distribution) or adaptive benchmarking strategies (model-specific question selection) demands additional theoretical and algorithmic advances.
• Data Augmentation: Integrating alternative data modalities (activations, multiple generations, in-context learning effects) may further enhance coreset utility.

Conclusion

This work demonstrates that efficient benchmarking for LLMs is well-captured by the intersection of information-theoretic feature selection (mRMR, especially MIQ) and kernel ridge regression. These techniques consistently outperform competitive alternatives across diverse metrics and benchmarks, particularly for binary-scored datasets. The proposed framework delivers compute-efficient, stable, and generalizable evaluation with minimal complexity, opening robust pathways for further research in scalable LLM benchmarking.

Figure 9: Optimization of MIQ objective during mRMR coreset construction directly enhances prediction error and ranking correlation.