How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions

Abstract: Neural scaling laws appraise data through dataset size, while the Vendi Score uses quantum entropy to measure dataset value. We show both that common neural-scaling-law objectives and the Vendi Score are submodular. We further show that the Vendi Score is a special case of a broader class of submodular objectives that we call matrix spectral functions. This also includes determinantal (DPP) objectives, as well as many others. We also introduce weakly matrix monotone functions and show how they lead to weakly submodular matrix spectral functions, yielding a broad family of practical objectives for data appraisal. We develop secular-equation-based updates that avoid repeated eigendecompositions during greedy optimization, reducing marginal-gain evaluation for $m$-dimensional embeddings by an $O(m)$ factor relative to oracle queries. This yields an average empirical speedup of about 35,000x, making direct optimization of the Vendi Score feasible on ImageNet-1K-scale datasets. Thus enabled, we compare how well several objectives predict the value of training subsets for held-out test performance under fixed-size, class-balanced, and fixed training-budget regimes, including the Vendi Score, DPPs, facility location, and three new matrix spectral variants. Across multiple datasets, facility location performs the best. Direct optimization also reveals that, while the Vendi Score is predictive over moderate score ranges, pushing the objective to higher values can make it a poor downstream performance proxy. We also find that uniformly at random fixed-size subsets, both unconstrained and class-balanced, are remarkably concentrated in both appraisal scores and held-out performance. Finally, we show that size, class balance, and training budget do not alone determine data value: even when controlling for these factors, performance ranges smoothly from good to bad.

• The paper shows that submodular valuation unifies neural scaling laws and the Vendi Score as special cases of matrix spectral functions.
• The authors introduce an efficient secular update method achieving 35×–220× speedups for optimizing dataset subsets on large-scale data.
• Empirical results indicate that the facility location surrogate best predicts downstream performance compared to size-based or entropy-based criteria.

Authoritative Analysis of "How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions" (2605.29448)

Introduction and Motivation

This work addresses quantifying the value of datasets in machine learning, focusing on both theoretical appraisal functions and their scalable optimization. While neural scaling laws typically reduce data value to dataset size, the paper critiques this naive proxy, emphasizing that intrinsic sample diversity and structure significantly modulate learning efficacy—an observation not captured by size-only models.

Two primary data valuation paradigms are compared: neural scaling-law objectives and the recently proposed Vendi Score, the latter grounded in the Von Neumann (quantum) entropy of the spectral distribution of dataset embeddings. The work rigorously characterizes these functions as submodular set functions and proceeds to generalize both as special cases of a broad class termed matrix spectral functions. Operator monotonicity and its weak analogs are developed to establish submodularity or weak submodularity in this context.

Scaling to large datasets is realized via a secular equation-based rank-1 update for eigenvalues, massively accelerating greedy optimization for matrix spectral criteria. The empirics interrogate how well different appraisal objectives predict downstream task performance relative to subset size, class balance, and compute—all controlled experimentally. Notably, empirical findings contradict some prior assessments regarding the Vendi Score and highlight the facility location function as the most predictive surrogate among those tested.

Theoretical Contributions

Unification and Submodularity of Data Valuation Functions

The authors provide a unified framework for data valuation based on set functions defined over subsets $X$ of a ground dataset $V$. They demonstrate:

• Neural scaling laws (e.g., Chinchilla, cluster-based, and epoch-based laws) are (strictly) submodular when cast as set functions, as their data value measures are concave in cardinality $|X|$, leveraging standard results from submodular analysis. This aligns their optimization properties with those of classical submodular objectives.
• The Vendi Score, defined as $f(X) = -\operatorname{tr}(B_X \log B_X)$ where $B_X$ is a PSD kernel matrix over $X$, is also submodular as $-\phi'(x)$ (with $\phi(x) = -x\log x$) is matrix monotone. This connects von Neumann entropy directly to submodularity theory, generalizing prior results for DPPs.
• Matrix spectral functions are introduced: any $f(X) = \operatorname{tr}[\phi(B_X)]$ where $\phi$'s negative derivative is matrix monotone yields a (strongly) submodular set function. This encompasses log-determinant (DPP), the Vendi Score, and other spectral surrogates.
• The authors define weakly matrix monotone functions, extending classical operator monotonicity. If $V$0 is only $V$1-weakly matrix monotone, $V$2 gains a $V$3-weak submodularity property. This relaxes theoretical requirements, allowing approximation guarantees for broader function classes, including some that empirically perform better.

Efficient Greedy Optimization via Secular Updates

A key algorithmic advance is an $V$4-per-query method for evaluating the marginal gain of adding a sample, based on efficient eigenvalue updates using the secular equation for rank-1 matrix perturbations. This avoids full eigendecomposition at every greedy step, yielding empirical speedups of $V$535×–220× on ImageNet-scale data for optimizing matrix spectral objectives.

Figure 1: Empirical speedup of the secular equation method over naive oracle calls for greedy maximization of the log Vendi score, as a function of dataset size $V$6, with feature dimension $V$7.

Empirical Evaluation

Experimental Design

• Dataset: Primarily ImageNet-1K, with extensions to 20 Newsgroups and Airbnb duplicate images.
• Selection Objectives: Direct (greedy) maximization/minimization for facility location, Vendi Score, DPP, and several other matrix spectral variants. Both fixed-size and class-balanced constraints are considered.
• Surrogates vs. True Performance: Each objective's value on a selected subset is compared to the held-out test accuracy of a ResNet-18 trained on that subset.

Key Findings

• Size Is a Poor Proxy for Data Value: Subsets with smaller cardinality and higher facility location scores consistently outperform larger, lower-score subsets—even under matched class balance and compute budgets.

Figure 2: Test accuracy versus appraisal score and size, showing that facility location is a better predictor than Vendi in both fixed-size and class-balanced regimes.

• Vendi Score Limitations: While Vendi Score correlates with performance over moderate value regimes, direct maximization to extreme values can select subsets that generalize poorly, at times yielding negative downstream correlation. This contradicts prior claims, which did not optimize Vendi Score directly over its full range.
• Facility Location Dominance: Among all appraisal functions, facility location exhibits the strongest, most monotonic correlation with downstream test accuracy across all controlled regimes (size, class balance, compute).

(Figure 3)

Figure 3: Comparison of Vendi Score and facility location as surrogates for downstream test accuracy across different subset selection mechanisms.

• Random Subsets Are Highly Concentrated: Random (even class-stratified) subsets exhibit much less variance in both appraisal score and test accuracy than deterministic optimization, exposing limited diversity in performance outcomes attainable by chance without direct optimization.
• Compute Alone Does Not Determine Value: For fixed compute budgets, having a higher-valued subset can yield better test accuracy with less compute than a lower-valued subset with more compute—further decoupling "worth" from mere resource allocation.

Practical and Theoretical Implications

• Subset selection for data pruning, core-set construction, and data purchasing should employ surrogates like facility location or suitably tuned matrix spectral functions rather than naive size-based or entropy-based criteria. The submodular (and weakly submodular) structure provides not only approximation guarantees under efficient greedy optimization but also reliably tracks latent utility for downstream generalization.
• Efficient secular-based greedy selection unlocks matrix spectral approaches at previously infeasible scales, enabling practical optimization of complex, global set functions beyond standard modular or additive policies.
• Random sampling, even with stratification, fails to span the full diversity of data values, underscoring the necessity for direct combinatorial optimization in data curation pipelines.
• The theoretical extension to weak submodularity broadens the design space of admissible spectral surrogates and clarifies why certain empirically effective but otherwise not fully submodular functions still yield near-optimal greedy solutions.
• Submodular data valuation provides immediate support for conditional and groupwise valuation mechanisms (e.g., Shapley-style credit assignment, or determining marginal value of new data given existing holdings).

Open Questions and Future Work

• Extending the secular equation approach for more general (nonlinear) kernels or kernelized spectral objectives, potentially enabling more expressive surrogates at scale.
• Systematic exploration and empirical tuning of weakly matrix monotone functions for empirical surrogate design beyond Shannon/Von Neumann entropy and log-determinant.
• Large-scale multivariate benchmarking across diverse modalities, tasks, and model architectures, particularly for new-found weakly submodular surrogates.
• Direct incorporation of more realistic data curation constraints, such as privacy restrictions, distributed data holdings, and dynamic (online) dataset valuation or acquisition.

Conclusion

This work establishes a rigorous unification of scaling laws, spectral entropy, and submodular optimization for dataset appraisal. The facility location function emerges as a robust and scalable proxy for data value, with efficient secular-based optimization making matrix spectral criteria practically viable at scale. The research exposes critical limitations of both size-based proxies and direct entropy- or Vendi-driven selection, advocating for submodular (or weakly submodular) objectives as the gold standard for practical and theoretical data valuation. The framework provides a solid foundation for future research in both scalable optimization and principled data-centric machine learning.