Iterative Magnitude Pruning (IMP)

• IMP is a network sparsification method that incrementally removes low-magnitude weights to discover 'winning tickets' that closely match dense model performance.
• It operates via repeated prune–train cycles using a binary mask for weight selection, achieving up to 90% pruning with minimal accuracy loss.
• IMP’s theoretical insights connect pruning with topological invariants and Renormalisation Group ideas, driving research on efficient model compression.

Iterative Magnitude Pruning (IMP) is an established network sparsification algorithm widely used in deep learning, valued for its simplicity, effectiveness, and conceptual transparency. Its central mechanism iteratively constructs a sparse subnetwork ("winning ticket") by gradually removing the least-magnitude weights and retraining the remaining parameters. IMP's empirical success in maintaining high post-compression accuracy across diverse architectures has made it the de facto baseline for the pruning community. Recent research has connected IMP to topological, physical, and optimization-theoretic principles, producing a rigorous and multifaceted understanding of its underlying efficacy (Balwani et al., 2022).

1. Algorithmic Structure and Standard Workflow

IMP operates via repeated prune–train cycles on a fully trained neural model $f(x;W)$, maintaining a binary mask $M \in \{0,1\}^{|W|}$ that designates active weights. Given a target sparsity $p$ and a pruning step size $\Delta$, the workflow is:

1. Train $f(x;W_t)$ for $T$ iterations to obtain weights $W_t$.
2. Identify the $\Delta\%$ of surviving weights with lowest $|W_t|$; set their corresponding mask entries to zero.
3. Apply the updated mask: $W_{t+1}=W_t \odot M$; optionally reset optimizer state.
4. Continue training and pruning until cumulative sparsity $\geq p$.

This procedure robustly finds sub-networks with $>90\%$ of weights pruned while preserving within $1\%$ accuracy of the dense baseline [Frankle & Carbin, 2018]. Mask selection is performed either globally or per-layer, often with a cubic schedule to adjust pruning rates over time (Park et al., 2021). Fine-tuning or "rewinding" the surviving weights to an early checkpoint in training can further stabilize performance (Frankle et al., 2019). IMP's precise update rule for the mask in each iteration is:

$$M^{(t+1)}_i = 1 \left\{ |W^{(t)}_i| \geq \tau^{(t)} \right\},$$

where $\tau^{(t)}$ is the magnitude threshold at which exactly the target fraction is pruned.

2. Topological and Graph-Theoretic Insights

Recent work frames IMP's efficacy through the lens of zeroth-order persistent homology (Balwani et al., 2022). Considering a neural layer as a weighted bipartite graph $G_k$ (edges as weights), a super-level set filtration by decreasing $|w|$ reduces the graph to its Maximum Spanning Tree (MST). The MST encodes all zeroth-order topological information—that is, the set of connected components (Betti number $\beta_0$).

The topologically critical compression ratio $n_T$ for a layer is:

$n_T = \frac{|W|}{|\mathrm{MST}(G)|},$

where $|W|$ is the number of weights and $|\mathrm{MST}(G)|$ the minimum number of edges to preserve $\beta_0$ connectivity. For fully connected layers with $m_k$ inputs and $n_k$ outputs, the bound is:

$n_T \geq \frac{m_k n_k}{m_k + n_k - 1}.$

Preserving MST edges at each pruning step guarantees retention of key structural information. Empirically, standard magnitude pruning already retains a significant fraction (40–60%) of MST edges at typical sparsities, well above random baseline (Balwani et al., 2022). This provides a principled explanation for IMP's stability and performance in extreme sparsity regimes.

3. Theoretical Connections to Renormalisation Group and Multiscale Structure

IMP admits a natural interpretation as a coarse-graining operator defined on parameter space, analogous to the Renormalisation Group (RG) transformations in physics (Hassan, 2023). The mask update and retraining steps generalize RG's projection and rescaling operations, respectively:

$\mathcal{T} = \mathcal{R} \circ \mathcal{M},$

where $\mathcal{M}$ masks weights and $\mathcal{R}$ retrains the survivors. Layer-wise “mass” under pruning,

$$M_i(n) = \frac{ \sum_{j \in \mathrm{layer}_i} |m_j(n)\cdot \theta_j(n)| }{ \sum_{k=1}^p |m_k(n)\cdot \theta_k(n)| },$$

obeys a recursion $M_i(n+1) = \lambda_i M_i(n)$, with RG exponent $\sigma_i = \log_{1/(1-x)}(\lambda_i)$, separating relevant and irrelevant directions.

Scaling behavior of test error near critical density $d_C$ exhibits power-law divergence,

$e(d) \sim (d_C - d)^{-\gamma},$

mirroring critical exponents in RG theory. Notably, masks ("winning tickets") discovered via IMP display universality and transferability across disparate problem domains if their relevant directions coincide.

4. Complexity, Variants, and Practical Considerations

IMP's principal computational bottleneck is the necessity of retraining between pruning steps, yielding a total cost scaling as $O(N \cdot E \cdot T_{\text{epoch}})$ where $N$ is the number of cycles, $E$ the epochs per cycle, and $T_{\text{epoch}}$ the training time per epoch (Saikumar et al., 1 Apr 2024). Complexity can be alleviated through early-stopping criteria as in Information Consistent Pruning (InfCoP), where retraining is terminated once layer-wise information or gradient flows are close to those of the dense model (Gharatappeh et al., 26 Jan 2025). InfCoP empirically achieves matching accuracy and sparsity with only 10–30% of the training time.

A further variant is Topological IMP (T-IMP), which augments mask selection by mandating preservation of MST edges, filling additional slots by magnitude as needed (Balwani et al., 2022). T-IMP ensures zeroth-order topological invariants are perfectly retained but incurs moderate additional computational cost (per-layer MST computations in $O(E \log V)$).

Recent advancements include the use of multi-particle averaging (SWAMP) to produce flatter minima and improved generalization, without increasing inference cost (Choi et al., 2023), and model-averaging ensembles (Sparse Model Soups) to exploit parallel retraining with shared masks (Zimmer et al., 2023).

5. Empirical Performance, Robustness, and Universality

IMP demonstrates robust accuracy maintenance under aggressive sparsification. For typical vision datasets and architectures (Saikumar et al., 1 Apr 2024):

Architecture	Dataset	90% Sparsity	95%	98%	99.3%
VGG-16	CIFAR-10	91.61%	90.20%	90.04%	88.35%
ResNet-18	CIFAR-100	64.44%	63.81%	61.94%	61.08%

In federated or edge settings (IIoT), IMP yields communication-efficient models with maintained or even improved accuracy compared to baseline and one-shot pruning methods, with pronounced improvements at high compression ratios (Khan et al., 21 Mar 2024).

IMP is notably robust to random sign perturbations in weights under Learning Rate Rewinding (LRR); its masks are less coupled to parameter initialization than one-shot or fine-tuning approaches (Gadhikar et al., 29 Feb 2024). Universality emerges in cross-domain transfer of masks, with RG exponents predicting success across tasks (Hassan, 2023).

6. Geometric, Loss-Landscape, and Inductive-Bias Effects

IMP minima after sequential pruning reside in distinct loss-landscape valleys, each within the same connected sublevel set (Saleem et al., 22 Mar 2024). Hessian spectra at these minima reveal broader ("flatter") loss basins than baseline projections from previous levels, supporting improved generalization. The second-order Taylor argument justifies the magnitude selection rule: pruning minimal $|w_i|$ causes minimal quadratic loss increase.

Inductive biases induced by IMP are quantifiable. On natural images, IMP increases the non-Gaussianity of hidden-layer preactivations, iteratively favoring the emergence of localized receptive fields—a property inherited from biological vision and convolutional networks (Redman et al., 9 Dec 2024). Empirical measurements (kurtosis, cavity method) show IMP prunes precisely those weights whose removal maximally amplifies non-Gaussian features, establishing a feedback loop promoting localization.

7. Open Problems and Directions

Outstanding areas include:

• Extension of homology-driven pruning to higher-order features ($\beta_1$, $\beta_2$, e.g. cycles/voids) for richer structural preservation (Balwani et al., 2022).
• Quantitative correlation between topological invariants and generalization/robustness.
• Development of global mask selection strategies beyond layerwise paradigms.
• Theory and practice of mask transfer and universality in cross-model and cross-domain settings.
• Combining topological criteria, efficient optimization (Bi-level methods), and ensemble averaging to further advance sparsification regimes.

IMP's conceptual clarity, topological rationalization, and extensibility ensure its ongoing centrality in deep-learning model compression methodologies (Balwani et al., 2022).