Is Data Shapley Not Better than Random in Data Selection? Ask NASH

Abstract: Data selection studies the problem of identifying high-quality subsets of training data. While some existing works have considered selecting the subset of data with top-$m$ Data Shapley or other semivalues as they account for the interaction among every subset of data, other works argue that Data Shapley can sometimes perform ineffectively in practice and select subsets that are no better than random. This raises the questions: (I) Are there certain "Shapley-informative" settings where Data Shapley consistently works well? (II) Can we strategically utilize these settings to select high-quality subsets consistently and efficiently? In this paper, we propose a novel data selection framework, NASH (Non-linear Aggregation of SHapley-informative components), which (I) decomposes the target utility function (e.g., validation accuracy) into simpler, Shapley-informative component functions, and selects data by optimizing an objective that (II) aggregates these components non-linearly. We demonstrate that NASH substantially boosts the effectiveness of Shapley/semivalue-based data selection with minimal additional runtime cost.

• The paper introduces NASH, a framework that decomposes complex utility functions into Shapley-informative components for improved data selection.
• It employs per-validation datum Shapley values and non-linear aggregation to overcome the limitations of conventional Data Shapley methods.
• Empirical results demonstrate that NASH consistently outperforms both Data Shapley and random selection across various datasets and models.

Authoritative Summary of "Is Data Shapley Not Better than Random in Data Selection? Ask NASH"

Problem Formulation and Limitations of Data Shapley

The paper addresses the problem of subset selection in machine learning, where one aims to select a subset of $m$ training data from a larger set under constraints such as budget or storage, maximizing a utility function (typically validation accuracy $u_V$). Conventional approaches often use Data Shapley values—a cooperative game-theoretic valuation method that scores data based on their marginal contributions aggregated over all possible subsets—selecting the top-$m$ highest valued data.

Despite theoretical guarantees (uniqueness in satisfying equitability axioms), recent empirical studies show Data Shapley can perform worse than random selection, particularly when the utility function is complex (e.g., $u_V$ averaging over many validation points with heterogeneous prediction difficulty). The paper formalizes the notion of "Shapley-informative" utility functions as those for which a subset’s sum of Data Shapley values reliably predicts its utility—showing that validation accuracy $u_V$ typically fails this criterion.

Figure 1: NASH decomposes complex validation accuracy $u_V$ into simpler Shapley-informative roles and aggregates their Shapley values non-linearly.

Analysis of Shapley-Informativeness and the Role Decomposition

The authors introduce a probabilistic definition of Shapley-informativeness: for utility function $u$, the sum of Shapley values in a subset should, with high probability, correlate to the true utility under some mapping. They mathematically demonstrate that arbitrary functions are not Shapley-informative due to high-dimensional kernel of the Shapley mapping, and empirically validate that $u_V$ is not Shapley-informative—showing indistinguishable Shapley values for data with complementary strengths but differing contributions to held-out validation points.

Figure 2: Toy model where all data have equal Shapley values, but actual utility differs by their coverage of distinct roles.

To overcome these limitations, the authors propose decomposing $u_V$ into individual prediction correctness functions $u_v$, one per validation datum. Under reasonable assumptions (consistent player: each training point’s marginal contributions are consistently good/bad), each $u_V$0 becomes nearly modular and provably Shapley-informative.

The NASH Framework: Non-Linear Aggregation

Building upon this decomposition, the authors present NASH (Non-linear Aggregation of SHapley-informative components), a framework that computes Shapley values for the $u_V$1 roles and aggregates them via a non-linear function inspired by learning curve phenomenology (exponential or power law). This non-linear aggregation ensures the interaction between data covering distinct roles is properly accounted for—a significant improvement over the linear sum that underpins top-$u_V$2 Shapley selection.

The NASH objective is maximized efficiently via a greedy algorithm, with negligible additional computational overhead relative to computing conventional Data Shapley values, because all $u_V$3 quantities can be obtained in the same pass.

Figure 3: NASH-selected subsets demonstrate improved coverage of poorly predicted validation roles compared to Data Shapley.

Numerical Experiments and Empirical Results

Experiments span standard data valuation tasks (e.g., Wind, Pol datasets with Logistic Regression), regression (Protein, Auction), and large-scale prompt-based finetuning (BERT, Llama-2-7B on sentiment and entailment datasets). NASH is compared to random selection, vanilla Data Shapley, and additional baselines (influence functions, TracIn, Representer points).

Figure 4: Validation accuracy $u_V$4 curves across selection ratios under various selection methods; NASH consistently outperforms Shapley and random.

NASH consistently outperforms Data Shapley and random selection in both homogeneous- and heterogeneous-quality datasets. Even in regimes where Data Shapley performs well (datasets with many corrupted/noisy points), NASH provides further improvements due to its handling of role coverage and non-linear aggregation.

Experiments also validate NASH's compatibility with a broad class of semivalues (e.g., Beta Shapley, Data Banzhaf), demonstrating that the framework is robust across different weighting schemes for coalition marginal contributions.

Figure 5: Data selection performance for WD-LR; NASH achieves higher validation accuracy across all selection sizes.

Theoretical and Practical Implications

The theoretical analysis establishes that Shapley-informative components exist for ML-induced utility functions under weak assumptions, and that complex objectives are not generally Shapley-informative. NASH leverages these insights, resulting in a consistently reliable data selection framework that circumvents the failures of top-$u_V$5 Shapley selection.

Practically, this has significant implications for subset selection in large-scale ML: NASH is computationally efficient and robust, compatible with improvements in Shapley value approximation, and generalizes across models, tasks, and datasets. The approach also provides interpretability—since each training point's strengths across roles can be directly analyzed.

Theoretically, the paper invites further study into the kernel structure of valuation operators and the design of utility functions amenable to subset optimization via cooperative game theory. The framework could inspire adaptive, diversity-aware, and richer selection criteria in future research.

Figure 6: Scatter of Shapley values for two training points shows major variability across validation roles, even when global Shapley values are equal.

Conclusion

"Is Data Shapley Not Better than Random in Data Selection? Ask NASH" (2605.10684) introduces a rigorous, practical framework that resolves longstanding questions about the effectiveness of Data Shapley in data selection. By decomposing complex utility functions into Shapley-informative components and employing non-linear aggregation, NASH consistently delivers superior subset selection performance while preserving the desirable cooperative game-theoretic properties. The approach's flexibility suggests broad applicability and potential for future extensions in principled data selection and valuation methodologies.