Pruning as a Game: Equilibrium-Driven Sparsification of Neural Networks (2512.22106v1)

Abstract: Neural network pruning is widely used to reduce model size and computational cost. Yet, most existing methods treat sparsity as an externally imposed constraint, enforced through heuristic importance scores or training-time regularization. In this work, we propose a fundamentally different perspective: pruning as an equilibrium outcome of strategic interaction among model components. We model parameter groups such as weights, neurons, or filters as players in a continuous non-cooperative game, where each player selects its level of participation in the network to balance contribution against redundancy and competition. Within this formulation, sparsity emerges naturally when continued participation becomes a dominated strategy at equilibrium. We analyze the resulting game and show that dominated players collapse to zero participation under mild conditions, providing a principled explanation for pruning behavior. Building on this insight, we derive a simple equilibrium-driven pruning algorithm that jointly updates network parameters and participation variables without relying on explicit importance scores. This work focuses on establishing a principled formulation and empirical validation of pruning as an equilibrium phenomenon, rather than exhaustive architectural or large-scale benchmarking. Experiments on standard benchmarks demonstrate that the proposed approach achieves competitive sparsity-accuracy trade-offs while offering an interpretable, theory-grounded alternative to existing pruning methods.

• The paper introduces a game-theoretic formulation where neurons act as strategic players, leading to emergent sparsity at equilibrium.
• It develops an algorithm that integrates weight updates with participation variable optimization using gradient descent and projected gradient ascent.
• Empirical results demonstrate that extreme sparsity (under 2% neurons retained) achieves over 91% test accuracy, validating the method's robustness.

Pruning as a Game: Equilibrium-Driven Sparsification of Neural Networks

Introduction and Motivation

The paper "Pruning as a Game: Equilibrium-Driven Sparsification of Neural Networks" (2512.22106) departs from the conventional view of neural network pruning as a post-hoc, centralized operation based on external importance scores or heuristic regularization. Instead, it proposes a game-theoretic formulation where parameter groups (e.g., neurons) are modeled as strategic players in a non-cooperative game. Here, each player's strategy, encoded as a participation variable, determines its level of involvement in the network. Sparsity, under this perspective, is not externally enforced but emerges at equilibrium when dominated strategies (redundant or non-contributive parameter groups) collapse to zero participation.

This approach offers a direct explanation for why sparsity arises in overparameterized networks, pointing to internal network competition and redundancy rather than solely to imposed constraints.

Game-Theoretic Framework for Pruning

Parameter groups within a network are assigned continuous participation variables $s_i \in [0, 1]$, acting as multiplicative gates on weights or neurons. The overall network computation thus becomes a function of both the original parameters and their participation levels, which are jointly optimized.

Each player's utility is designed to balance their marginal contribution to the loss with multiple forms of penalty:

• Benefit: The marginal decrease in loss attributed to the player, computed via the gradient inner product $$\langle \nabla_{\theta_i} \mathcal{L}, \theta_i \rangle$$.
• Cost: Includes an $\ell_2$-norm penalty, an $\ell_1$ penalty to promote sparsity, and a competition term discouraging redundancy (measured via correlations with other parameter groups).

The Nash equilibrium corresponds to a portfolio of participation values where no player can unilaterally improve its utility. Dominated strategies (subsets of parameter groups for which costs always outweigh benefits) optimally collapse to zero, thus being pruned.

Theoretical Analysis

The authors derive the best-response dynamics for participation updates, revealing a closed-form that resembles soft-thresholding commonly found in regularized optimization. Crucially, they show that:

• Players whose marginal benefits cannot overcome sparsity and competition costs will be driven to zero participation at equilibrium.
• The competition term (correlation with other active parameter groups) accentuates the pruning of redundant components.
• With appropriately chosen penalty coefficients (notably, a strong enough $\ell_2$ term), the equilibrium is both sparse and stable.

This framework unifies diverse pruning heuristics by explicating that typical approaches (magnitude pruning, gradient pruning, redundancy-based pruning) emerge as special cases under different utility specifications.

Equilibrium-Driven Pruning Algorithm

The implementation is straightforward: both network weights and participation variables are updated iteratively. Weights follow the usual gradient descent given the current participation mask, while participation variables undergo projected gradient ascent to maximize their utilities, constrained to $[0,1]$. All $s_i$ are initialized to one. After training, parameter groups with participation below a small threshold (e.g., 0.01) are pruned.

This intertwined optimization removes the need for explicit importance score computation or separate pruning-finetuning phases, and pruning emerges seamlessly during training.

Experimental Analysis

The empirical validation is conducted on an MLP architecture for MNIST, focusing on neuron-level pruning in two hidden layers. Multiple configurations of $\ell_1$ and $\ell_2$ cost coefficients are evaluated to probe the conditions under which sparsity emerges.

During training, the following behaviors are observed:

• Insufficient penalty: Lax cost configurations lead to dense equilibria; participation values hover modestly below one but do not collapse.
• Aggressive regularization: Strong $\ell_1$, $\ell_2$, or especially combined penalties induce rapid collapse of participation variables, yielding high sparsity with minimal loss in test accuracy.

  Figure 1: Training dynamics of equilibrium-driven pruning under different utility configurations. The four-panel visualization shows test accuracy, sparsity, mean participation, and active neurons over epochs. Stronger penalties trigger collapse of dominated strategies.

At convergence, the distribution of participation variables in effective pruning regimes is sharply bimodal: most neurons are either near-zero (pruned) or near-one (fully active), rather than suspended in intermediate states.

Figure 2: Distribution of neuron participation values at convergence. Bimodality aligns with near-binary equilibrium decisions for pruned versus retained neurons; dense settings yield unimodal distributions.

Quantitative results are particularly informative: with less than 2% of neurons retained, over 91% test accuracy is maintained—demonstrating that equilibrium dynamics are effective at locating redundant structure without explicit importance scoring.

Implications and Extensions

The equilibrium-driven perspective provides several insights relevant to theoretical and practical research:

• Unifying Heuristics: Many widely used pruning methods (magnitude, gradient, redundancy awareness) can be interpreted as instantiations of particular utility functions or cost pressures in this framework.
• End-to-End Sparsification: By integrating pruning into network training (rather than as a separate, post-hoc operation), the approach facilitates dynamic sparsification and potentially seamless adaptation to changing data or deployment constraints.
• Interpretability: The bimodal occupation of the participation variable space aligns with clear, interpretable decisions about neuron necessity, as opposed to threshold-based ambiguity.

Potential extensions include scaling to deep or structured architectures, exploring hierarchical or adaptive game formulations, and integrating with efficient hardware-aware regularization.

Conclusion

The equilibrium-driven sparsification framework reframes neural network pruning as an emergent outcome of intra-network competition mapped onto a strategic game. Both the theoretical analysis and empirical results, including markedly bimodal equilibrium participation and robust accuracy at extreme sparsities, validate that such an approach is not only sound but also practically viable. While this study focuses on controlled settings, further work is suggested to address the challenges of scaling and robustness in complex and large-scale scenarios.