Equilibrium-Driven Neural Network Sparsification

• Equilibrium-driven sparsification is a framework that naturally reduces model complexity by freezing redundant neural units once their dynamics stabilize.
• It employs methods such as NEq, game-theoretic pruning, and entropic minimization to achieve robust, structured sparsity with minimal accuracy loss.
• Empirical results demonstrate substantial FLOP reductions and interpretability improvements compared to traditional threshold-based pruning techniques.

Equilibrium-driven sparsification of neural networks refers to a suite of frameworks and algorithms in which the identification and removal (or freezing) of redundant or inactive model components—such as weights, neurons, or filters—is governed by notions of dynamical or game-theoretic equilibrium. Rather than enforcing sparsity via externally-imposed thresholds or purely heuristic criteria, these approaches drive sparsity as a natural outcome of stable or optimal behavior in either the learning dynamics or a formalized cost-benefit game among parameter groups. This principled perspective encompasses methods based on neuronal equilibrium, game-theoretic pruning, sparsification in equilibrium-propagation learning, and entropy-minimization for selection of active subnetworks.

1. Mathematical Definition and Principles of Equilibrium

Equilibrium-driven sparsification leverages the notion that certain elements of a neural model—parameters, neurons, or groups thereof—can be judged redundant or fully “learned” when their activity, utility, or dynamics settle into steady-state, i.e., equilibrium. The mechanisms for defining equilibrium depend on the framework.

Neuronal Equilibrium (NEq):

Let $y_{i,n,\xi}^{t}$ denote the $n$-th activation of neuron $i$ in response to sample $\xi$ at epoch $t$ on a held-out validation set $\Xi_{val}$. Define the normalized activation: $$\hat{y}_{i,n,\xi}^{t} = \frac{y_{i,n,\xi}^{t}}{ \| y_{i,\cdot,\cdot}^{t} \|_2 }$$ The inter-epoch cosine similarity is computed as: $$\varphi_{i}^{t} = \sum_{\xi\in \Xi_{val}} \sum_{n=1}^{N_i} \hat{y}_{i,n,\xi}^{t} \hat{y}_{i,n,\xi}^{t-1}$$ A neuron is at equilibrium if the velocity of this similarity,

$$v_{\Delta\varphi_i}^t = \Delta\varphi_{i}^t - \mu_{eq} v_{\Delta\varphi_i}^{t-1}$$

with $$\Delta\varphi_{i}^t = \varphi_{i}^t - \varphi_{i}^{t-1}$$, becomes sufficiently small: $|v_{\Delta\varphi_i}^t| < \epsilon$ (Bragagnolo et al., 2022).

Game-Theoretic Equilibrium:

Parameter groups (players) select a participation level $s_i\in[0,1]$ and compete in a continuous noncooperative game. Each player's utility function combines its benefit to loss reduction and its cost due to magnitude penalties and pairwise competition: $$U_i(s_i, s_{-i}) = B_i(s_i, s_{-i}) - C_i(s_i, s_{-i})$$ At Nash equilibrium, participation for dominated parameters collapses to zero ($s_i^* = 0$), i.e., the unique maximizer of $U_i$ over $[0,1]$ is $0$ (Shah et al., 26 Dec 2025).

Entropic Equilibrium:

A soft selection probability vector $w \in \Delta^D$ is optimized to minimize the sum of mean-squared error and an entropy penalty: $$\min_{w\in\Delta^D, \Lambda} \epsilon_w \sum_{d=1}^D w_d \log w_d + \epsilon_{l_2} \|\Lambda\|_F^2 + \mathrm{MSE}$$ When $\epsilon_w < 0$, the optimizer favors low-entropy (i.e., sparse) distributions; at equilibrium, most $w_d$ approach zero and the resulting subnetwork is sparse (Barisin et al., 6 Apr 2024).

2. Algorithmic Procedures and Pseudocode

NEq Procedure

Executed per training epoch:

1. Train model on the training set for epoch $t$.
2. For each neuron $i$, compute activation statistics on $\Xi_{val}$, normalize, evaluate $\varphi_{i}^t$, $\Delta\varphi_{i}^t$, and update $v_{\Delta\varphi_i}^t$.
3. Freeze neuron $i$ (mask gradients and parameter updates) for next epoch if $|v_{\Delta\varphi_i}^t| < \epsilon$.
4. Unfreeze if not; in the following epoch, only unfrozen neurons participate in updates. (Bragagnolo et al., 2022)

Game-Theoretic Pruning

At each iteration:

1. Update parameters $\theta$: $$\theta \leftarrow \theta - \eta_\theta \nabla_\theta \mathcal{L}(\theta, s)$$.
2. For each $i$, update participation: $$s_i \leftarrow \mathrm{Proj}_{[0,1]}\left( s_i + \eta_s \, \partial_{s_i} U_i(s) \right)$$
3. After $T$ iterations, prune all $i$ with $s_i < \varepsilon$ (set $s_i = 0$, remove $\theta_i$). (Shah et al., 26 Dec 2025)

Entropic Layer-wise Pruning

For each layer:

1. Alternate ridge regression for $\Lambda$ (fixed $w$) and convex optimization in $w$ (fixed $\Lambda$).
2. After convergence, prune all channels $d$ with $w_d < 10^{-6}$. This two-step, monotonically decreasing procedure fits the equilibrium over feature-selection weights (Barisin et al., 6 Apr 2024).

3. Structured Sparsification and Its Operational Impact

Equilibrium-driven sparsification results in structured patterns of inactivation at the granularity dictated by the method—entire neurons, filters, or even channels. For NEq, when a fraction $f(t)$ of neurons reach equilibrium, the effective backward-pass FLOPs per iteration are reduced by $S(t) = 1-P_{active}(t)/P_{total}$, directly proportionate to the number of frozen elements (Bragagnolo et al., 2022). Unlike unstructured weight pruning, structured freezing or removal minimizes runtime overhead and can be implemented without additional indexing.

In game-theoretic frameworks, the contraction mapping guarantees a unique fixed point for participation scores when the redundancy penalty $\eta$ is not too large (Shah et al., 26 Dec 2025). This means the resulting sparsity patterns are robust and interpretable: full/inactive bimodality is commonly observed in participation histograms, with few elements left in an ambiguous intermediate state.

4. Empirical Validation and Comparative Results

NEq Performance:

Reported on standard image tasks and models:

Dataset/Model	Baseline FLOPs	NEq FLOPs	FLOPs Reduction	Baseline Acc	NEq Acc
CIFAR-10/ResNet-32	138.94M	84.81M	–38.96%	92.85%	92.96%
ImageNet-1K/ResNet-18	3.64G	2.80G	–23.08%	69.90%	69.62%
ImageNet-1K/Swin-B	30.28G	10.78G	–64.39%	84.71%	84.35%
COCO/DeepLabv3	305.06G	217.29G	–28.77%	67.71%	67.22%

The structured freezing outperformed random-freeze baselines; even as the active parameter count dropped by $>40\%$ in mature epochs, test performance did not degrade (Bragagnolo et al., 2022).

Game-Theoretic Pruning (MNIST MLP):

Config.	Test Acc.	Sparsity	% Neurons Kept
Very High Beta	96.64%	0.00%	100%
Extreme Beta	91.15%	95.18%	4.82%
L1 Sparsity Strong	89.57%	98.31%	1.69%
L1+L2 Combined	91.54%	98.05%	1.95%

Very high L1/L2 penalties drive up to $98\%$ neuron sparsity with only modest loss in accuracy. Intermediate participation values are unstable; only near-0 or near-1 values persist at equilibrium (Shah et al., 26 Dec 2025).

Entropic Pruning:

• LeNet (MNIST): 55.9–83.3% parameters pruned, $<$0.6% test accuracy drop, full recovery after modest fine-tuning.
• VGG-16/CIFAR-10: 88.7% parameters pruned vs. 65–92.7% for other methods; 93.87% accuracy (baseline 94.08%) (Barisin et al., 6 Apr 2024).

5. Theoretical Context and Model Dynamics

The equilibrium view for sparsification ties to dynamical systems and convex optimization theory:

• NEq: Neuron output mappings are funneled into basins of attraction; freezing is only enacted when the cosine-similarity velocity stabilizes (Bragagnolo et al., 2022).
• Game-theoretic: Sparsity is endogenous, and the utility function unifies magnitude, saliency, and redundancy-based heuristics. Collapse of dominated strategies to $s_i=0$ follows from the concavity of $U_i$ in $s_i$.
• Equilibrium Propagation: In “Equilibrium Propagation for Complete Directed Neural Networks,” an $\ell_1$-regularized Hebbian update with probabilistic pruning is provably stable under Lyapunov analysis; the pruning step does not harm local attractivity (Farinha et al., 2020).
• Entropic Equilibrium: By reframing the NP-hard $\ell_0$ selection as an entropy-minimization convex program, entropic sparsification achieves sublinear scaling (in #channels/layers/patches), with convergence to stationary points guaranteed per convex optimization theory (Barisin et al., 6 Apr 2024).

6. Limitations, Extensions, and Future Directions

Empirical evidence suggests equilibrium-driven schemes are robust across architectures and tasks but several limitations and avenues remain:

• Current NEq implementations treat neurons in isolation; co-equilibration across groups, layers, or blocks may enable further compute reductions (Bragagnolo et al., 2022).
• The impact of optimizer and schedule tuning on rate of convergence to equilibrium is under-explored, though it is suggested that tailored schemes could amplify sparsity benefits.
• While NEq and related methods focus on training-time sparsity, extending the logic to inference—permanently pruning or quantizing equilibrated units—remains promising and is identified as future work (Bragagnolo et al., 2022).
• In game-theoretic pruning, the framework unifies many pruning heuristics and provides interpretability but large-scale architectural benchmarking is less explored (Shah et al., 26 Dec 2025).
• Entropic relaxation is currently applied layer-wise with post-pruning fine-tuning required for best accuracy; automatic extension to end-to-end joint minimization and multi-branch networks is suggested (Barisin et al., 6 Apr 2024).
• In equilibrium-propagation frameworks, application beyond small directed-graph testbeds to deep architectures and real-world tasks is not yet demonstrated (Farinha et al., 2020).

7. Relation to Broader Sparsification Methods

Equilibrium-driven sparsification algorithms stand in contrast to exogenous-score-based pruning (magnitude, gradient, entropy) and structured unstructured sparsity imposition. A key distinction is the endogenous, dynamics-driven mechanism for element inactivation or pruning, resulting in more interpretable, theoretically grounded, and sometimes computationally favorable sparsification patterns. These frameworks are compatible with but fundamentally differ from methodologies where sparsity is imposed via fixed constraints or after-the-fact rewinding and retraining (Bragagnolo et al., 2022, Shah et al., 26 Dec 2025, Barisin et al., 6 Apr 2024, Farinha et al., 2020).