---
title: Shapley Value Pruning
url: https://www.emergentmind.com/topics/shapley-value-pruning
type: topic
---

# Shapley Value Pruning

Shapley value pruning is a suite of methods applying the Shapley value from cooperative game theory to identify and remove superfluous, redundant, or even harmful components from machine learning systems. By evaluating the average marginal utility of each component (e.g., neuron, filter, interaction, layer, attention head, or data point) across all possible coalition subsets, Shapley value pruning assigns a theoretically justified importance score that accounts for both synergy and redundancy. Across neural architectures and data modalities, this approach yields principled, data-driven rankings for pruning and denoising; these rankings are supported by multiple sampling and amortized approximation strategies that scale to modern network sizes and data volumes. Empirical results demonstrate that Shapley-based pruning consistently improves efficiency–accuracy trade-offs relative to conventional magnitude- or gradient-based heuristics, especially in low-data and noise-sensitive regimes.

## 1. Shapley Value Formalism and General Pruning Framework

At the core of Shapley value pruning is the cooperative game formulation. Given a set $N$ of components and a value function $v:2^N\to\mathbb{R}$ quantifying system utility for each subset, the Shapley value $\phi_i$ for component $i\in N$ is defined as:
$$
\phi_i = \sum_{S\subseteq N\setminus\{i\}} \frac{|S|! (|N|-|S|-1)!}{|N|!} \bigl[v(S\cup\{i\}) - v(S)\bigr]
$$
where $v(S\cup\{i\}) - v(S)$ is the marginal contribution of $i$ to $S$. This definition ensures four key axioms: efficiency ($\sum_i\phi_i = v(N)-v(\emptyset)$), symmetry (identically contributing players are equally rewarded), null-player (zero impact components receive zero value), and additivity (linearity in $v$).

In pruning applications, the "players" can represent neural units, model parameters, features, data samples, or other structural submodules. The value function $v$ is closely linked to downstream metrics—typically accuracy, negative loss, or some validation objective measured on a (possibly masked/pruned) subsystem. Shapley values thus provide a global, context-aware ranking resilient to redundancy and negative contributions.

## 2. Algorithms and Approximations for Shapley Value Estimation

For practical deployments, computing exact Shapley values is infeasible for nontrivial $|N|$ due to exponential coalition space. Multiple approximation protocols have been introduced:

- **Partial Shapley (leave-$k$-out):** Restricts the coalition sum to subsets of fixed size $k$; reduces cost to $\mathcal{O}(N \binom{N-1}{k})$ [2407.15875].
- **Monte Carlo permutation sampling:** Samples $k$ random permutations $\pi$ of $N$, estimates $\phi_i$ as the mean marginal increase when $i$ joins the prefix of $\pi$ [2006.01795, 2210.05709, 2407.15875].
- **Weighted least squares regression (Shapley kernel):** Fits a linear model to $v(S)$ observed for sampled $S$, using the Shapley kernel as weights [2407.15875, 2507.20460].
- **Sliding-Window/Local Approximations:** Restricts Shapley computation to a local window around each component, e.g., neighborhood of adjacent layers in a transformer to capture only meaningful interactions [2505.01731].
- **Surrogate neural networks:** Trains a parametric surrogate (e.g., MLP) to amortize $v(S)$ prediction for rapid Shapley estimation over coalitions [2505.22057, 2602.07804].
- **Antithetic permutation pairing:** Combines permutation orderings and their reversals to reduce variance and bias [2510.01663].

These methods yield efficient, scalable estimates with quantifiable approximation bias/unbiasedness, enabling end-to-end pruning in networks with hundreds or thousands of prunable units.

## 3. Instantiations Across Model Classes and Data Modalities

### Neural Network Compression

In CNN and MLP channel pruning, each channel/unit is a player, and $v(S)$ is the network's validation accuracy or negative loss when only $S$ is active. Shapley pruning demonstrably outperforms magnitude-based and Taylor-gradient heuristics across LeNet, VGG, and ResNet architectures on both MNIST and CIFAR-10, achieving up to $153\times$ compression and minimal increase in test error [2407.15875]. Similarly, in structured pruning under no- or low-fine-tuning, Shapley-based rankings result in up to $5$–$15\%$ higher retained top-1 accuracy than alternatives [2006.01795].

### Recommender Systems and Data Pruning

In the context of recommender systems, Shapley Value-driven Valuation (SVV) treats each user–item interaction as a player, with $v(S)$ corresponding to the reduction in squared error of a denoising autoencoder reconstructed only from $S$. SVV uses amortized FastSHAP estimation, then prunes the bottom $k\%$ of interactions in estimated $\phi$. This approach yields $3$–$7\%$ gains in Recall@10 and NDCG@10, recovers up to $64\%$ of injected long-tail noise, and reveals interpretable distinctions between pivotal and negligible behavioral records [2505.22057].

### Large Language Models: Layer and Head Pruning

Layer-wise Shapley value pruning in LLMs assigns value to transformer layers, with $v(S) = 1/\mathrm{PPL}(S)$. Non-uniform sparsity (SV-NUP) based on estimated $\phi$ preserves critical layers and prunes redundant ones, yielding $18\%$–$20\%$ relative PPL reductions at $70\%$ sparsity in LLaMA-7B/13B models over SparseGPT [2505.01731]. Surrogate-assisted Shapley in LLMs further leverages a lightweight MLP to predict $v(S)$ for arbitrarily masked layer combinations, aggregating stratified Monte Carlo marginal contributions for efficient large-scale application [2602.07804].

Head-level Shapley pruning in multilingual transformers isolates and excises attention heads with negative impact on per-language performance, identifying language-specific interference. Pruning with Shapley rankings can produce $+3.78$ accuracy points in XNLI-Chinese and up to $+12.4\%$ relative improvement in UDPOS-Chinese, without retraining [2210.05709].

### Non-standard Architectures and Shift-Invariance

In Kolmogorov–Arnold Networks (KANs), where functional weights are spline maps and coordinate-invariant pruning is essential, Shapley value–based "ShapKAN" scores are uniquely shift-invariant, outperforming magnitude scoring on symbolic regression, neuron-rank stability under affine shifts, and post-pruning RMSE [2510.01663].

### Graph Neural Network Sparsification

Shapley-based edge importance in GNNs assigns both positive and negative contributions to edges in the computational graph of each node. Aggregating these edge-wise $\phi$ values globally allows pruning of misleading and redundant edges, reducing inference MACs by up to $65\%$ while incurring less than $2\%$ accuracy loss at high sparsity thresholds, outperforming gradient- and MI-based baselines [2507.20460].

## 4. Theoretical Properties, Guarantees, and Limitations

Shapley value pruning satisfies a set of axiomatic optimality and fairness properties: efficiency (total attribution matches actual performance delta), null-player (units with no marginal value receive zero attribution), symmetry (exchangeable units tie), and additivity. This framework uniquely minimizes expected harm for a given number of pruned components and is robust to redundancy, negative synergy, and shared responsibility among correlated units or features [2006.01795, 1610.02080]. 

However, practical limitations include computational overhead for large $N$ (addressed by approximation schemes), possible redistribution of importance under strong dependencies among pruned units, and architecture-specific value function choices. In heavily correlated input or model spaces, Shapley ranks reflect shared, not solely marginal, effect; removal of one unit may cause importance redistribution among survivors [1610.02080].

## 5. Empirical Evaluation and Comparative Analysis

Experiments consistently demonstrate that Shapley value pruning achieves state-of-the-art accuracy–efficiency trade-offs across tasks and domains. In CNNs, Shapley-based schemes match or surpass oracle ranking within $90\%$ of maximum possible overlap with optimal prune sets [2407.15875]. LLM layer- and head-level pruning via Shapley outperforms magnitude, random, and outlier-weighted layerwise allocations by up to $20\%$ relative in model perplexity and $3$–$4$ points in zero-shot accuracy [2505.01731, 2602.07804]. In recommenders, SVV yields both higher overlap with ground-truth injected noise and, in some cases, test accuracy that even exceeds that of "clean" (uncontaminated) training, reflecting benefits for tail/balance correction [2505.22057]. In GNNs, Shapley-pruned graphs retain superior accuracy at fixed sparsity relative to MI-, gradient-, or lottery-ticket-based approaches [2507.20460].

## 6. Interpretability, Robustness, and Practical Guidance

Shapley value pruning offers intrinsic interpretability, with per-unit (or per-data, per-edge) explanations linked directly to marginal system utility. For example, low-$\phi$ (but nominally "important") high-rating CDs can be flagged as negligible for collaborative filtering, revealing the difference between frequency and effect [2505.22057]. Aggregated attributions enable identification of adversarial, redundant, or negative-synergy components; in ensemble games, adversarial models systematically receive near-zero Shapley value and can be excluded for improved robustness [2101.02153].

Recommended best practices involve modest sample budgets ($K=5$–10 for MC-based methods), aggregation of attributions to avoid over-pruning occasional but sometimes critical units (e.g., via $\mathrm{mean} + 2\,\mathrm{std}$), and Monte Carlo or kernel regression for tractable yet precise approximation. Sliding-window and surrogate strategies are essential for very large LLMs. In low-data regimes where fine-tuning is unfeasible, Shapley methods consistently minimize post-prune accuracy loss.

## 7. Extensions and Outlook

The Shapley value pruning paradigm has been generalized to feature selection under dependency [1610.02080], data pruning for denoising [2505.22057], hierarchical and groupwise pruning (Owen value), and even symbolic recoverability in KAN/other structured networks [2510.01663]. Ongoing research focuses on amortized networks for hyper-efficient Shapley estimation, higher-order interaction attributions, and better value functions for nascent architectures. Future directions include extension to configuration games, adaptive sampling of coalitions, and combining Shapley value with alternative cooperative game indices for specialized compression or robustness objectives.

---

**Key references:**
- Shapley Value-driven Data Pruning for Recommender Systems [2505.22057]
- Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models [2505.01731]
- Shapley Pruning for Neural Network Compression [2407.15875]
- Structured Network Pruning via Shapley Values [2006.01795]
- Shift-Invariant Attribute Scoring for Kolmogorov-Arnold Networks via Shapley Value [2510.01663]
- Shapley-Value-Based Graph Sparsification for GNN Inference [2507.20460]
- Shapley Head Pruning in Multilingual Transformers [2210.05709]
- On Shapley value for measuring importance of dependent inputs [1610.02080]
- The Shapley Value in Machine Learning [2202.05594]
- Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for LLMs [2602.07804]
- The Shapley Value of Classifiers in Ensemble Games [2101.02153]

Source: https://www.emergentmind.com/topics/shapley-value-pruning