SparseEval: Efficient LLM Benchmarking

• SparseEval is an evaluation framework that reduces the number of benchmark items by optimizing a sparse weighting problem with a learned nonlinear aggregator.
• It employs cosine similarity and spectral clustering to identify redundant items and select a small set of representative anchors for efficient benchmarking.
• Empirical results show SparseEval achieves lower MAE and higher Kendall’s τ compared to traditional methods, significantly cutting inference costs.

SparseEval is an efficient evaluation framework for LLMs that formulates benchmark reduction as a sparse optimization problem on a model–item performance matrix. In its canonical form, it assumes that benchmark items are redundant, that a small subset of representative items can serve as anchors, and that full-benchmark accuracy can be recovered from anchor responses by learning a nonlinear aggregation map. The framework was introduced as a method for estimating benchmark scores and model rankings from a small item subset while preserving low estimation error and high Kendall’s $\tau$, thereby reducing the inference cost of large-scale benchmarking (Zhang et al., 8 Feb 2026).

1. Problem formulation and structural assumptions

SparseEval begins from a binary model–item score matrix

$S \in \{-1,1\}^{m \times n},$

where $m$ is the number of models, $n$ is the number of benchmark items, and

$$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$

Each column $S_{:,j}$ is the response pattern of all models on one item, and each row is the itemwise outcome vector of a model. The full benchmark score vector is expressed as

$\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$

so that $\mu_i$ is the mean benchmark accuracy of model $i$ (Zhang et al., 8 Feb 2026).

The method is motivated by empirical regularities in the item–item similarity matrix. For items $j$ and $S \in \{-1,1\}^{m \times n},$0, SparseEval uses cosine similarity

$S \in \{-1,1\}^{m \times n},$1

collects these values into an item–item similarity matrix, and applies spectral clustering. The reported structure consists of strong diagonal blocks together with non-trivial inter-cluster similarity. This supports the claim that many items are redundant and that a small number of anchors can act as representative points for efficient benchmarking (Zhang et al., 8 Feb 2026).

The resulting optimization target is a sparse weighting problem with a learned aggregation function: $S \in \{-1,1\}^{m \times n},$2 Here $S \in \{-1,1\}^{m \times n},$3 is a sparse item-weight vector with at most $S \in \{-1,1\}^{m \times n},$4 nonzeros, $S \in \{-1,1\}^{m \times n},$5 is the anchor budget, and $S \in \{-1,1\}^{m \times n},$6 aggregates the weighted anchor responses into an estimate of the full benchmark score. If $S \in \{-1,1\}^{m \times n},$7 is restricted to be linear, the problem reduces to

$S \in \{-1,1\}^{m \times n},$8

which describes earlier subset-selection approaches such as TinyBenchmark and TailoredBench. SparseEval departs from that setting by learning a nonlinear $S \in \{-1,1\}^{m \times n},$9 with an MLP (Zhang et al., 8 Feb 2026).

2. Anchors, nonlinear aggregation, and sparse optimization

In SparseEval, anchors are the items with nonzero entries in $m$0. For a new model, evaluation is carried out only on these anchors, and the resulting anchor response vector is passed through a learned predictor to estimate the full-benchmark score. The framework states that it is the first to adopt gradient descent to optimize anchor weights in this benchmarking setting and to combine this with iterative anchor refinement (Zhang et al., 8 Feb 2026).

Given a fixed anchor set and a training score matrix $m$1, the learned aggregator is trained with

$m$2

The implementation used for LLM benchmarks is a 4-layer MLP. The input dimension is the anchor budget $m$3, the output is a single scalar benchmark-score estimate per model, and optimization uses gradient descent with learning rate $m$4 (Zhang et al., 8 Feb 2026).

Anchor initialization exploits the observed item clustering. For benchmarks with strong intra-cluster similarity, SparseEval uses $m$5-means in item space and selects cluster centroids as anchors. For benchmarks with weaker clustering, such as MMLU, random initialization performs comparably or better. The procedure therefore evaluates both $m$6-means and random selection on a small validation set and retains the better initialization (Zhang et al., 8 Feb 2026).

The method is explicitly hard-budgeted: sparsity is enforced by maintaining exactly $m$7 anchors rather than by introducing an $m$8 penalty. This makes the operational cost at test time proportional to $m$9, while the learned MLP contributes negligible additional computation relative to model inference on benchmark items (Zhang et al., 8 Feb 2026).

3. Iterative refinement: AIS and CIS

SparseEval refines anchors through two task-aware scores: the Anchor Importance Score (AIS), which measures the influence of current anchors on the loss, and the Candidate Importance Score (CIS), which measures the potential value of non-anchor items. Both are defined relative to validation residuals

$n$0

These residuals encode modelwise over- and under-estimation errors under the current sparse proxy (Zhang et al., 8 Feb 2026).

For a non-anchor item $n$1, CIS is

$n$2

A large CIS indicates strong alignment between that item’s response pattern and the current residual vector, so the item is informative about the proxy’s present failures. For an anchor item $n$3, AIS is defined from the gradient norm of the loss with respect to the anchor input,

$n$4

A small AIS marks a weak anchor; a large AIS marks an anchor whose perturbation would materially change the objective (Zhang et al., 8 Feb 2026).

The refinement loop proceeds by training a proxy MLP on the current anchor set, computing residuals, evaluating AIS on anchors and CIS on non-anchors, then performing one swap: $n$5 followed by

$n$6

The anchor budget remains fixed because every addition is paired with one removal. The reported configuration uses $n$7 refinement steps and anchor sizes in $n$8 (Zhang et al., 8 Feb 2026).

The paper also supplies linear-case intuition. If $n$9 denotes the best achievable reconstruction error under anchor set $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$0, then $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$1 implies $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$2. A second proposition states that replacing the least useful anchor with the most useful candidate does not increase error after re-optimization, and strictly decreases it when the incoming candidate has larger residual correlation and is not in the span of the current anchor set. This provides a formal justification for the AIS/CIS swap rule in the linear approximation (Zhang et al., 8 Feb 2026).

4. Empirical performance and comparative position

SparseEval is evaluated on ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande using model–item matrices derived from Open LLM Leaderboard v2 and extended to approximately 5000 models. The experimental split samples 200 models for validation and 200 for test, with the remainder used for training. Performance is reported with mean absolute error (MAE) on benchmark score estimation and Kendall’s $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$3 on induced model rankings (Zhang et al., 8 Feb 2026).

Across all reported anchor counts, SparseEval is described as consistently achieving the lowest MAE and highest Kendall’s $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$4 among the compared methods: Anchor Points, gp-IRT from TinyBenchmarks, and TailoredBench. The following examples at 100 anchors illustrate the scale of the reported differences.

Benchmark	Method	MAE / Kendall’s $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$5
ARC	Anchor Points	$$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$6 / $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$7
ARC	gp-IRT	$$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$8 / $$S_{i,j} = \begin{cases} 1 & \text{if model } i \text{ answers item } j \text{ correctly}\ -1 & \text{if model } i \text{ answers item } j \text{ incorrectly.} \end{cases}$$9
ARC	TailoredBench	$S_{:,j}$0 / $S_{:,j}$1
ARC	SparseEval	$S_{:,j}$2 / $S_{:,j}$3
GSM8K	gp-IRT	$S_{:,j}$4 / $S_{:,j}$5
GSM8K	TailoredBench	$S_{:,j}$6 / $S_{:,j}$7
GSM8K	SparseEval	$S_{:,j}$8 / $S_{:,j}$9

The paper further states that with 100 anchors, SparseEval on ARC performs similarly to baselines that require about 500 anchors, implying a greater than $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$0 reduction in the number of benchmark items that must be evaluated per model. This is operationally important because inference cost at deployment time is dominated by the number of items presented to the model (Zhang et al., 8 Feb 2026).

The ablations identify three central effects. First, $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$1-means initialization usually improves over random selection, especially at small anchor counts, but the subsequent refinement stage improves over both. Second, replacing the MLP with a linear model degrades performance substantially, most visibly at small $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$2, indicating that the mapping from anchor scores to full benchmark scores is not well captured by a purely linear estimator. Third, reducing the number of training models degrades accuracy, but SparseEval reportedly remains stronger than gp-IRT and still achieves MAE below $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$3 and $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$4 on HellaSwag with only $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$5 of the training data (Zhang et al., 8 Feb 2026).

A separate hold-out evaluation on DeepSeek-family models—covering model scales from $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$6B to $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$7B and dense, MoE, coder, math, and general variants—shows improved robustness relative to gp-IRT and TailoredBench. The reported average MAEs are $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$8 for TailoredBench, $\mu = S W_a,\qquad W_a = \frac{1}{n}\mathbf{1}_n,$9 for gp-IRT, and $\mu_i$0 for SparseEval; the corresponding maximum MAEs are $\mu_i$1, $\mu_i$2, and $\mu_i$3 (Zhang et al., 8 Feb 2026).

5. Operational use, assumptions, and limitations

SparseEval is intended for repeated evaluation scenarios in which a historical model–item matrix is already available. The training cost is amortized across future models: once anchors and the final MLP are learned, evaluating a new model requires inference on only the anchor set, followed by a negligible forward pass through the predictor. If full evaluation costs $\mu_i$4 item inferences, SparseEval reduces this to $\mu_i$5 with $\mu_i$6 (Zhang et al., 8 Feb 2026).

The framework predicts aggregate benchmark scores rather than per-item or per-category behaviour. This is a material modeling choice: the target of inference is

$\mu_i$7

and evaluation uses

$\mu_i$8

with MAE

$\mu_i$9

Ranking fidelity is measured by Kendall’s $i$0, so the method is specifically optimized for score recovery and leaderboard order preservation rather than detailed behavioural diagnostics (Zhang et al., 8 Feb 2026).

Its main assumptions are also its main limitations. It requires a sufficiently large historical matrix of full benchmark results; without such a corpus, the nonlinear predictor cannot be learned. The paper notes that gradient-based learning may underperform with very small numbers of training models. It also assumes a degree of stationarity: the mapping from anchor performance to full benchmark performance may drift as new architectures or capabilities emerge, which implies that retraining or refresh may be necessary. Finally, although AIS/CIS refinement reduces sensitivity to initialization, the final anchor set still depends on choices such as initialization scheme, anchor count, number of refinement rounds, and MLP architecture (Zhang et al., 8 Feb 2026).

The framework’s practical significance lies in its combination of low deployment cost, strong ranking fidelity, and faster training relative to IRT-based alternatives. The reported training advantage over gp-IRT is attributed to the use of a simple MLP and standard gradient descent instead of expensive IRT fitting. A plausible implication is that SparseEval is particularly attractive for benchmark maintenance workflows in which many models must be re-scored frequently under stable benchmark definitions (Zhang et al., 8 Feb 2026).

6. Broader “SparseEval” usage across sparse-method research

Although SparseEval is the title of a specific LLM-evaluation method, the term also appears as a broader evaluative perspective in several areas of sparsity research. In these usages, the shared theme is not a single algorithm but a family of criteria for judging sparse approximations: fidelity, sparsity, structural parameters, computational efficiency, and downstream utility.

In matrix sparsification, “SparseEval” is associated with operator-level accuracy. The entrywise sampling framework for numerically sparse matrices evaluates a sparsifier $i$1 by spectral error

$i$2

sparsity through $i$3, and near-optimality through lower bounds. It identifies numerical sparsity $i$4 and stable rank $i$5 as structural parameters governing the intrinsic difficulty of sparsification (Braverman et al., 2020). This usage treats sparse evaluation as a theory-backed tradeoff between approximation quality and complexity.

In sparse non-negative matrix factorization, the same perspective emphasizes joint tracking of reconstruction quality, clustering or topic quality, runtime, and peak memory. The enforced-sparsity ALS method explicitly evaluates relative residual

$i$6

relative approximation error

$i$7

factor nonzero counts, and topic-wise clustering accuracy, while also recording maximum nnz during training as a memory metric (Gavin et al., 2015). This broadens sparse evaluation from static sparsity to time-varying computational behaviour.

Sparse tensor computation provides an explicitly benchmark-oriented variant of the idea. A parallel sparse tensor benchmark suite defines representative kernels—TEW, TS, TTV, TTM, and MTTKRP—over COO and HiCOO formats on CPUs and GPUs, and evaluates them with GFLOPS, operational intensity, and Roofline efficiency (Li et al., 2020). Here a SparseEval-style framework is one in which sparse data layouts, workload realism, and hardware-aware performance models are jointly standardized.

In Bayesian multidimensional scaling, sparse evaluation becomes a question of how much of an $i$8 dissimilarity matrix can be omitted without sacrificing inferential quality. Sparse BMDS measures mean squared distance error, Hellinger distance between posterior summaries, and ESS per hour under banded or landmark subsampling, obtaining approximately $i$9-fold, $j$0-fold, and $j$1-fold speedups with negligible loss of accuracy at 500, 1000, and 5000 points when using 50 bands or landmarks (Sheth et al., 2024). The emphasis shifts from algebraic sparsity to subsampled likelihood design.

Sparse autoencoder analysis adds a geometric extension to this broader usage. SAEMA evaluates sparse representations through rank profiles of SSPD matrices, Average Geodesic Distance, local intrinsic dimensionality, Betti $j$2, MSTW, Procrustes disparity, and AEDP, and reports a significant causal relationship between separability of local representations and reconstruction performance (Sun et al., 28 May 2025). This suggests a further generalization of SparseEval in which topology and manifold structure become first-class sparse evaluation objects.

Taken together, these works suggest that “SparseEval” names both a concrete LLM benchmarking method and a wider research motif: sparse surrogates are not judged solely by how few entries, items, or features they retain, but by whether they preserve the task-relevant structure measured by rigorous accuracy, efficiency, and downstream-effect criteria (Zhang et al., 8 Feb 2026, Braverman et al., 2020, Gavin et al., 2015, Li et al., 2020, Sheth et al., 2024, Sun et al., 28 May 2025).