Pruning Neural Networks at Initialization: Why are We Missing the Mark? (2009.08576v2)

Abstract: Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

Pruning Neural Networks at Initialization: A Review

The paper "Pruning Neural Networks at Initialization: Why Are We Missing the Mark?" by Frankle, Dziugaite, Roy, and Carbin addresses the efficacy and challenges associated with pruning neural networks at the very onset of the training process. The investigation focuses on popular methods such as SNIP, GraSP, SynFlow, and magnitude pruning, aiming to ascertain their viability compared to conventional pruning strategies executed post-training. Despite advances in pruning techniques that exceed the performance of random pruning, the paper reveals a consistent lag in accuracy compared to methods applied after training.

Key Findings

1. Comparative Performance of Pruning Methods:
   • The analysis shows that no single method emerges as the state-of-the-art (SOTA) across different network architectures and sparsities. While SNIP often performed well, magnitude pruning also proved surprisingly competitive with more complex heuristics.
   • At matching sparsities, which are key benchmarks where magnitude pruning post-training mirrors the accuracy of the unpruned network, the early pruning methods could not match this performance level.
   • SynFlow, despite its theoretical underpinnings ensuring maximum connectivity, exhibits limitations such as neuron collapse, where entire neurons or channels are disproportionately pruned.
2. Ablation Studies and Sensitivity Analysis:
   • The authors conducted various ablation studies, including random shuffling and reinitialization of weights in the pruned networks. The findings suggest that these early pruning methods are not sensitive to specific weight configurations or initial values—a stark contrast to pruning methods applied after training.
   • This insensitivity raises questions about the efficacy of existing heuristics in determining critical weights for network performance at initialization.
3. Challenges Specific to Initialization:
   • The authors speculate that these behaviors might be intrinsic to the challenges of pruning at initialization rather than deficiencies in the heuristics themselves. Further experiments after some network training showed improved accuracy and increased sensitivity to ablations, particularly for SNIP and magnitude pruning.
   • Pruning at initialization may not utilize sufficient signal for making informed decisions about which weights are essential, thus questioning the overall feasibility of this strategy for maintaining high accuracy.

Theoretical and Practical Implications

The findings set a realistic baseline for what can be achieved with current methodologies for pruning at initialization. The need for deeper theoretical understanding of network optimization properties at this stage becomes evident. The performance delta between initialization and post-training pruning emphasizes the necessity for innovative proxies capable of capturing essential training dynamics from the outset.

From a practical perspective, while the promise of reduced computational costs during training through early pruning is alluring, trade-offs in model accuracy remain an unsolved issue. For methods like SynFlow and GraSP to fulfill their potential, they must evolve to harness deeper network insights that extend beyond basic gradient or magnitude information.

Future Directions

The results suggest several avenues for future research:

• Development of Novel Signals: Exploring alternative metrics that better capture which weights are critical at initialization.
• Dynamic Pruning Techniques: Leveraging dynamic changes in sparsity patterns, potentially borrowing from approaches seen in dynamic sparse training paradigms, to adjust decisions as more information becomes available during early epochs.
• Broader Implications and Cost Trade-offs: Evaluating methods that incorporate both computational savings and accuracy considerations, possibly using adaptive training approaches that acknowledge the non-linear cost structures associated with different hardware architectures.

In summary, while progress has been made in pruning at initialization, this paper elucidates the substantial distance yet to be covered. The authors provide a foundational critique, serving as a stimulus for advancing both the theoretical understanding and practical implementations of early-stage pruning methods. This endeavor remains crucial in the context of scaling neural networks efficiently amidst growing computational constraints.