Progressive Sparsification Schedule

• Progressive sparsification schedule is a method that incrementally removes network parameters to avoid abrupt performance degradation.
• It employs techniques such as temperature annealing, grow-and-prune cycles, and regularization ramping to achieve high sparsity with maintained accuracy.
• The approach is versatile, supporting online, federated, and combinatorial applications for efficient model compression and network control.

A progressive sparsification schedule is a temporally structured protocol for gradually inducing sparsity in neural networks or combinatorial structures by incrementally increasing the number of zero-valued parameters or removing elements through a prescribed sequence of steps. These schedules are central to modern pruning, compression, and sparse-approximation techniques in deep learning, federated learning, online/anytime regimes, and network control. Progressive approaches avoid the pitfalls of one-shot pruning by leveraging controlled transitions that favor model stability, recovery, and accuracy retention.

1. Mathematical Foundations of Progressive Sparsification

The core motivation for progressive sparsification is to avoid catastrophic degradation from abrupt removal of network parameters. The dominant mathematical strategy is to introduce a sparsity-inducing term—typically an $\ell_0$ penalty or discrete mask—in the optimization objective. Directly solving the combinatorial problem

$\min_{w}~L(f(;w)) + \lambda\,\|w\|_0$

is not tractable at scale, so surrogate relaxations are employed, e.g., continuous gates via sigmoid or hard-concrete parameterizations (Savarese et al., 2019), or stochastic masks (Yuan et al., 2020). The progressive schedule is then realized by annealing relevant hyperparameters (e.g., mask temperature, regularization strength) or iteratively updating layerwise sparsity targets, with schedule-specific mechanisms for enforcing stability and recovery.

2. Progressive Sparsification in Deep Neural Network Pruning

Several paradigms implement progressive sparsification schedules in deep learning, targeting either static architectures (pruning) or dynamically changing ones (grow-and-prune, dynamic sparsification).

• Temperature Annealing (Continuous Sparsification): The mask $m_i = \sigma(\beta s_i)$ flexibly interpolates between fully dense ($\beta=1$) and binary gating ($\beta \to \infty$) regimes. The schedule

$\beta^{(t)} = (\beta^{(T)})^{t/T}$

transitions masks from “soft” to “hard” progressively, yielding rapid sparsity increase in early epochs and stabilization thereafter. Empirically, 80–90% global sparsity is reached within a single round, with diminishing increments in subsequent rounds (Savarese et al., 2019).

• Grow-and-Prune Cycles: Scheduled grow-and-prune (GaP) divides a network into $\kappa$ partitions. At each step, one partition is densified (“grown”) and another re-pruned, cycling through all partitions. The global sparsity oscillates piecewise-constantly, ensuring all weights receive multiple growth and pruning opportunities, mitigating suboptimal mask selections (Ma et al., 2021).
• Progressive Regularization Ramping: In structured continuous sparsification, both the sparsity-inducing Lagrange multiplier $\lambda$ and the mask temperature are ramped linearly or exponentially over the first $T_{\text{sch}}$ training steps, with no early collapse if $\lambda$ grows sufficiently slowly (Yuan et al., 2020).

3. Schedule Designs: Parameterizations and Hyperparameters

Robust progressive schedules are parameterized to balance rapid compression, recoverability, and final accuracy:

Schedule Type	Key Parameter(s)	Functional Form	Notable Defaults
Sigmoid/Temperature annealing	$\beta^{(T)}$, $R$, $T$	$\beta^{(t)} = (\beta^{(T)})^{t/T}$	$\beta^{(T)} = 200$, $\lambda=10^{-8}$, $R=3$–5, $T=85$
Exponential/Power Law (FedSparsify)	$n$, $F$, $S_0$, $S_T$	$$s_t = S_T + (S_0 - S_T)(1-\frac{F\lfloor t/F\rfloor-t_0}{T-t_0})^n$$	$n=3$, $F=1$, $S_0=0$, $S_T=0.9,0.95,0.99$
Stepwise Layerwise (Directed-Evolution)	$\Delta s_\ell$	$$S_\ell^{(t)} = \min(S_\ell^{(t-1)}+\Delta s_\ell, S_\ell^{\text{final}})$$	$\Delta s_\ell=0.05$, $K=120$ trials, per-layer control
Partition cycling (GaP)	$\kappa$, $K$, $T$, $r$	Piecewise over partitions, see Section 2	$\kappa$ ≈ blocks, $K=5$–$8\times\kappa$, $r=0.8, 0.9$

The hyperparameter controlling “hardening” or sparsification rate (e.g., $\beta$, $n$) is pivotal: set too low, masks never binarize; set too high, learning becomes unstable or irreversible.

4. Progressive Schedules in Specialized Regimes and Architectures

• Anytime/Online Progressive Pruning (APP): For data streams, the APP protocol applies pruning at each megabatch $t$ using a precomputed schedule $s(t)=0.8^{\delta_t}$, with a linearly spaced exponent $\delta_t$ and practical base $0.8$ to provide mild early pruning and more aggressive late reduction. This aligns capacity adaptation with continued learning, as opposed to linear or one-shot schedules, which induce brittle under- or over-pruning (Misra et al., 2022).
• Federated Progressive Sparsification (FedSparsify): In decentralized settings, FedSparsify incrementally raises the sparsity target across federated rounds via an exponential schedule. Pruning is performed globally or locally at each client, with synchronization via purging masks and majority-vote aggregation. Hyperparameters on pruning frequency and exponent $n$ trade off accuracy retention and speed of sparsity induction (Stripelis et al., 2022).
• Dynamic Spatial Sparsification: In transformer/CNN vision models, prediction modules progressively prune tokens or spatial locations at multiple points via hierarchical geometric or arithmetic schedules (e.g., $\rho^{(s)} = \rho^{s}$). This input-adaptive, staged approach preserves dense computation for important features while routing others through lightweight computations, maintaining final map structure and accuracy (Rao et al., 2022).

5. Empirical Performance, Failure Modes, and Ablations

Progressive sparsification schedules exhibit several empirical trends:

• Rapid early sparsity gains (e.g., $\sim$80% after 20–30 epochs in VGG-16/ResNet-20 on CIFAR-10),
• Diminishing per-round improvements due to early saturation of negative gates or layerwise mask convergence,
• Superior end accuracies to static methods (e.g., continuous sparsification surpasses iterative magnitude pruning at both efficiency and test error for VGG/CIFAR-10 and ResNet-50/ImageNet) (Savarese et al., 2019),
• Robustness to moderate variations in regularization strength or pruning step size, with mask initialization the critical determinant for hitting precise sparsity targets,
• Nonmonotonic generalization gap evolution in online regimes, with potential double-descent effects as APP regularizes overfitting peaks by late-stage pruning (Misra et al., 2022).

Ablation studies consistently show that too aggressive pruning early (e.g., high exponent $n$, high $\Delta s_\ell$) leads to accuracy degradation, while too slow schedules incur extra training cost with minimal sparsity gain. Full replay buffers and schedule restarts per pruning step are essential for stable performance in online/pruning-at-each-megabatch settings.

6. Applications Beyond Deep Learning: Sparse Scheduling in Circuit Switches

Progressive sparsification schedules also govern allocation in combinatorial systems such as circuit switches. The Birkhoff and Birkhoff$^+$ algorithms progressively construct an $\epsilon$-approximate decomposition of a demand matrix $T$ into a sparse sum of permutations. At each step, the scheme greedily selects a new permutation and duration to maximize error decrease, yielding total step count $k = O(\log(1/\epsilon))$. This enables operators to balance schedule sparsity (to minimize reconfiguration delay) against total approximation error for network throughput (Valls et al., 2020).

7. Guidelines and Best Practices for Implementation

Key recommendations for implementing effective progressive sparsification schedules include:

• For deep networks, use $\beta^{(T)} \in [150,250]$, $\lambda=10^{-8}$, mask initializations in $[-0.3,+0.3]$, and 3–5 continuation rounds for robustness and maximal compression (Savarese et al., 2019).
• In federated settings, favor small pruning frequency ($F=1$) and moderate exponent $n=3$ for smoothed transitions (Stripelis et al., 2022).
• When applying APP in online learning, base scheduling on $0.8^{\delta_t}$ with full replay buffers, adjusting granularity according to megabatch counts and final desired sparsity (Misra et al., 2022).
• In grow-and-prune schemes, partition according to network modularity (e.g., residual blocks), cycle multiple times per partition, and fine-tune after final pruning; for global sparsity, avoid explicit time interpolation—let partition cycling enforce exploration (Ma et al., 2021).
• For structured continuous sparsification, ramp regularization $\lambda$ linearly over an extended schedule to prevent premature collapse and maximize attainable sparsity (Yuan et al., 2020).

These progressive methods systematically interpolate between dense initialization and sparse optimality, enabling efficient, robust, and highly sparse models or schedules across neural and combinatorial architectures.