Iterative Pruning and Resetting

• Iterative Pruning and Resetting is a cyclic process that trains neural networks by iteratively pruning low-magnitude weights and resetting surviving connections to their initial state.
• It reveals 'winning ticket' subnetworks that maintain high accuracy even at 80-90% sparsity, thereby enabling efficient model compression and robust generalization.
• Empirical results demonstrate that this method not only enhances hardware deployment efficiency but also stabilizes training by navigating toward flatter, more generalizable loss minima.

Iterative Pruning and Resetting (IPR) is a methodology for identifying and training sparse subnetworks within over-parameterized neural networks. Central to this approach is the recurring cycle of training, criterion-based pruning, and weight resetting (rewinding) to a fixed reference point, most commonly the initial random initialization or an early training checkpoint. This paradigm not only underpins key empirical results in the Lottery Ticket Hypothesis—namely, the existence of "winning ticket" subnetworks—but has also provided the foundation for high-fidelity model compression, efficient hardware deployment, and a deeper understanding of loss landscape navigation and generalization in deep learning models (Saleem et al., 2024, Frankle et al., 2019, Liu et al., 2024, Paganini et al., 2020).

1. Algorithmic Foundations and Pseudocode

The iterative pruning and resetting framework is formally characterized as a cyclic process consisting of three distinct phases: (1) training the network (or subnetwork) to a specified convergence criterion; (2) pruning a prescribed fraction of parameters based on an importance metric (typically magnitude or post-activation); (3) resetting the surviving parameters to a reference state, usually the original random initialization or a fixed "rewind" checkpoint (Saleem et al., 2024, Liu et al., 2024). This process is repeated for multiple rounds until a desired global sparsity is achieved.

Canonical IPR Pseudocode (Magnitude-based, Zero-Rewind):

w = w0 m = ones_like(w0) while fraction_zeros(m) < S*: w = Train(f(·; w ⊙ m), D, epochs=E) # Score surviving weights by magnitude s = abs(w) ⊙ m threshold = percentile(s[s>0], p * 100) m = m * (s > threshold) w = w0 ⊙ m # Reset surviving weights to initialization return w, m

Pruning thresholds are determined via quantile selection to excise the smallest-magnitude parameters, and the mask update is conducted such that only parameters exceeding the threshold remain active. In the general formulation, the weight reset step can utilize either the original initialization or a checkpoint corresponding to early-phase training ("rewinding"), with the critical insight that moderate rewinding (after a brief warm-up period) facilitates subnetwork stability in large/deep architectures (Frankle et al., 2019).

2. Theoretical Rationale and Loss Landscape Analysis

Iterative pruning and resetting exploits the geometry of the solution landscape in over-parameterized neural networks, where low-loss, high-generalization minima are frequently sequestered in narrow valleys of the weight space—not readily accessible via vanilla stochastic optimization. Each pruning cycle excises high-curvature directions (small-magnitude weights), thus exposing previously hidden, but potentially broader and flatter minima when optimizing over the lower-dimensional subspace. The critical role of resetting is to reinitialize surviving parameters so that stochastic gradient descent (SGD) explores these new basins effectively, avoiding entrapment in the shrunken basin corresponding to the previously pruned network (Saleem et al., 2024).

Quantitative trajectory analysis employs geometric metrics:

• Hessian spectrum volume proxy: The sum of log-eigenvalues of the Hessian reveals "sharpness" (high values, tight minima) or "flatness" (many small eigenvalues, wide minima) of the current solution basin.
• Radial volume estimation: Monte Carlo sampling of the radius within which the increase in loss remains below a small threshold quantifies basin breadth.
• Angular alignment: Cosine similarity between successive IPR solutions is used to monitor the solution trajectory in weight space.

Empirical evaluation indicates that after retraining from each rewind point, the Hessian spectra shift in a cyclic “shrink–grow–shrink” pattern, signifying that IPR progressively uncovers flatter minima that may be inaccessible in the dense regime (Saleem et al., 2024).

3. Empirical Performance, Generalization, and Mask Dynamics

Key experimental phenomena systematically observed across vision models and datasets include:

• High-sparsity resilience: On canonical datasets (CIFAR-10, Tiny ImageNet) and architectures (ResNet-20, VGG-16), up to 80–90% of weights can be pruned iteratively with negligible loss in test accuracy. Only beyond this regime does accuracy degrade, correlating with emergent loss barriers and reduced connectivity (Liu et al., 2024, Saleem et al., 2024).
• Distributional dynamics: After each pruning/reset cycle, the histogram of surviving weights becomes increasingly heavy-tailed, with larger-magnitude parameters concentrated in the support of the active subnetwork.
• Mask alignment: Pruning masks derived from iterative magnitude-based procedures (with resetting) are highly consistent between iterations (inter-level cosine > 0.95), whereas one-shot pruning or random pruning leads to solution divergence (cos θ < 0.8) (Saleem et al., 2024).
• Block-structured emergence: Even when applying strictly unstructured (elementwise, $L_1$-based) pruning, the mask patterns revealed by IPR often cluster along contiguous rows or columns in convolutional layers, closely approximating structured pruning and yielding latent hardware efficiency (Paganini et al., 2020).
• Weight stability: The average change in surviving weights between pruning rounds, denoted $\Delta^{(t)}$, is minimized with unstructured magnitude-based pruning plus rewinding, and lower $\Delta^{(t)}$ correlates with higher achievable sparsity at constant test accuracy (Paganini et al., 2020).

4. The Role of Resetting and "Lottery Ticket" Mechanisms

Resetting to the original initialization or a suitable early checkpoint is essential for three reasons (Saleem et al., 2024, Frankle et al., 2019):

1. Unveiling accessible minima: By pruning away high-curvature, low-importance directions, IPR reveals sparser, flatter regions of the loss landscape that require specific initial conditions for SGD to converge efficiently.
2. Escaping poor optima: Simply fine-tuning or training on a pruned mask without resetting causes optimization to remain trapped in shriveled, suboptimal basins.
3. Iterative advantage over one-shot: High-sparsity one-shot pruning yields excessive loss barriers, while gradual, staged pruning keeps the basin transitions smooth and discoverable with reset SGD.

Resetting thus enables identification of "winning tickets"—subnetworks which, when retrained in isolation from their original initial values, match or even exceed the accuracy of the full dense model in an equivalent number of epochs (Liu et al., 2024). Empirically, rewinding to a small, nonzero fraction of training (0.1–7% of total iterations) is optimal for stabilizing subnetworks in deep networks (Frankle et al., 2019).

5. Extensions, Variants, and Comparative Analyses

IPR methodologies have inspired a rich array of extensions:

• DropNet generalizes IPR beyond weights to nodes and filters, using average absolute post-activation as the salience metric. Up to 90% of nodes/filters can be pruned with negligible accuracy loss, and resetting can employ either the original or freshly re-randomized weights, with little difference up to moderate sparsity (Min et al., 2022).
• AutoSparse and pruning-at-initialization (PaI): IPR trajectories are used to train lightweight neural scoring functions that emulate the survival probability of parameters under IRP, permitting efficient one-shot pruning. AutoSparse, for instance, recovers nearly all of IRP's performance with a single forward/backward pass and minor architecture-specific adaptation (Liu et al., 2024).

Comparative analyses consistently show that:

• Iterative pruning with resetting outperforms one-shot or post hoc pruning schemes at high sparsity.
• The subnetwork topology is not unique: distinct pruning criteria discover sparse masks with low Jaccard overlap but comparable accuracy, and ensemble methods can further improve final performance (Paganini et al., 2020).

Empirical summary tables illustrate these phenomena, e.g., for CIFAR-10:

Sparsity %	IRP (rewind)	PaI: SNIP	PaI: GraSP	PaI: Random	AutoSparse
90%	92.8	91.5	91.0	89.2	92.6
95%	89.3	86.2	86.8	83.5	88.9
98%	82.7	75.4	77.1	70.2	81.9

6. Subnetwork Stability, Mask Metrics, and Broader Significance

Pruned-and-reset subnetworks exhibit greater stability to both pruning and data-order stochasticity than randomly-pruned or non-reset counterparts. Two stability metrics are quantified (Frankle et al., 2019):

• Stability to pruning: $$S_{prune} = \mathbb{E}_u \| m \odot W_T^{full}(u) - m \odot W_T^{sub}(u) \|_2$$, measuring endpoint agreement with the full model.
• Stability to data order: $$S_{data} = \mathbb{E}_{u,u'} \| m \odot W_T^{sub}(u) - m \odot W_T^{sub}(u') \|_2$$.

Subnetworks discovered via IPR not only deliver competitive accuracy but also demonstrate enhanced robustness to gradient noise and serve as reliable building blocks for ensembles. Multiple distinct “winning tickets” with orthogonal subspaces can coexist within a single random initialization (Paganini et al., 2020).

A plausible implication is that in modern over-parameterized networks, IPR does not merely discover a single "lottery ticket," but instead reveals a multiply-connected, high-dimensional landscape of sparse, robust, and highly-adaptive subnetworks. The iterative pruning and resetting paradigm, by retaining critical initialization structure while progressively eliminating redundant parameters, plays a central mechanistic role in this emergence.

---

Principal Sources:

• (Saleem et al., 2024) Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning
• (Liu et al., 2024) Learning effective pruning at initialization from iterative pruning
• (Paganini et al., 2020) On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• (Min et al., 2022) DropNet: Reducing Neural Network Complexity via Iterative Pruning
• (Frankle et al., 2019) Stabilizing the Lottery Ticket Hypothesis