Dynamic Sparse Training Patterns

• Dynamic sparse training is a set of algorithms that jointly optimize network weights and binary masks, maintaining a fixed global sparsity through continuous rewiring.
• It uses periodic mask updates, such as magnitude-based pruning and gradient-based regrowth, to explore optimal sparse architectures while preserving high accuracy.
• Structured variants, including N:M and channel-level patterns, align with hardware capabilities to deliver significant speedups in training and inference.

Dynamic sparse training pattern refers to a family of algorithms and frameworks in which both the parameters (weights) and the nonzero connectivity pattern (mask) of a neural network are jointly learned throughout the entire training process, subject to a fixed global sparsity budget. Unlike post-hoc or static pruning approaches that operate on a pre-trained dense model or maintain an unchanging sparse structure, dynamic sparse training (DST) continually rewires the active set of connections, enabling the network to explore a broader set of sparse architectures and often achieve dense-level accuracy at significantly reduced parameter counts and computational costs.

1. Core Principles of Dynamic Sparse Training

Dynamic sparse training algorithms maintain two core objects: the parameter tensor $\theta$ (weights) and an associated binary mask $M \in \{0,1\}^{N}$ of the same shape, determining which parameters are active (nonzero) at each training step. For a fixed target sparsity $S$ (e.g., 90%), the set of nonzeros $\|M\|_0 = (1-S)N$ remains constant, but $M$ evolves dynamically according to structured rules. The essential loop comprises:

• Weight updates: Only active parameters ($\theta_s = \theta \odot M$) are updated via gradient-based optimizers.
• Mask rewiring: At scheduled intervals, a fraction $p$ of current connections are pruned (often by magnitude) and the same number of new connections regrown (either randomly or via a saliency metric, e.g., gradient magnitude), keeping the parameter budget fixed.
• Continuous joint-learning: The evolving mask and weight set are co-optimized from random initialization ("sparse-to-sparse") without dense pretraining (Liu et al., 2021).

This sparse-to-sparse regime stands in contrast to dense-to-sparse pipelines, offering lower training FLOPs, reduced memory footprint, and—when combined with structured patterns—practical speedups on modern hardware (Liu et al., 2021, Lasby et al., 2023).

2. Algorithmic Variants and Mask Update Paradigms

Multiple DST regimes have been developed, differing mainly in how drop and grow candidate connections are chosen, how sparsity is managed layer-wise, and their compatibility with model architectures.

• Magnitude-based drop & gradient-based grow: The RigL and sparse evolutionary training (SET) algorithms prune the smallest-magnitude active weights and regrow an equal number of connections with the highest gradient magnitude among the zeros—enabling rapid exploration of salient sparse topologies (Hu et al., 2024, Ullah et al., 2024).
• Random regrowth: In certain domains (notably RNNs), randomly regrowing new connections rather than using gradient criteria yields superior results, as it avoids topological traps induced by greedy gradient-based growth (Liu et al., 2021).
• Layerwise or block-wise budget allocation: Sparsity can be distributed non-uniformly across layers (e.g., via ERK initialization or polynomial schedules), and modern variants integrate this with structured channel or block granularity (Yin et al., 2023).
• Structured sparsity constraints: Recent DST approaches enforce N:M fine-grained (Fang et al., 2023), constant fan-in (Lasby et al., 2023), diagonal (Tyagi et al., 13 Jun 2025), and block/channel (Dietrich et al., 2021, Yin et al., 2023, Tyagi et al., 16 Oct 2025) sparsity, unlocking real-world speedup on accelerators.
• Neural architecture modularization: Some variants introduce mask updates at gate-level (LSTM, GRU), neuron-level (ablation in ViTs), or higher-level units (channel, block), employing specialized redistribution or ablation strategies to maximize regularization and hardware efficiency (Liu et al., 2021, Lasby et al., 2023, Tyagi et al., 13 Jun 2025, Yin et al., 2023).

The table below illustrates typical mask-mutation schedules:

Method	Drop Criterion	Grow Criterion	Regrowth Granularity
RigL/SET	Magnitude	Gradient magnitude	Weight
Selfish-RNN	Magnitude	Random	Weight & LSTM gate
Channel-aware	UMM / WS (channel stats)	Gradient (blockwise)	Channel
SRigL	Magnitude (structured block)	Gradient (blockwise)	Constant-fan-in/N:M block
DynaDiag	TopK diagonal offset	Differentiable soft-K	Diagonal

3. Mathematical Framework and Mask Optimization

DST optimization encompasses both discrete mask search and continuous weight updates, often realized as a joint or alternating process. Typical formulations include:

• Induced Mask via Thresholding: For tensor $\theta^l$ and threshold vector $\theta_{thresh}^l$, mask entries are defined as $$M^l_{ij} = \mathbb{I}\{|\theta^l_{ij}| > \theta_{thresh}^l\}$$ (Liu et al., 2020).
• Discrete-Continuous Sparse Connectivity Search: The DST-EE framework poses mask selection as a joint optimization

$M \in \{0,1\}^{N}$0

alternating SGD steps on active weights with mask updates via a UCB-style acquisition function blending exploitation (gradient) and exploration (activation count) (Huang et al., 2022).

• Structured Parameterization: For block, channel, or N:M patterns, mask optimization involves group-wise norm metrics, blockwise thresholding, or permutation-augmented layouts to maximize expressivity while adhering to the structure (Lasby et al., 2023, Tyagi et al., 13 Jun 2025, Tyagi et al., 16 Oct 2025).

Convergence and generalization guarantees have been established under mild smoothness and variance assumptions for the discrete-continuous optimization loop, showing sublinear convergence of the average gradient norm to zero (Huang et al., 2022).

4. Hardware Efficiency, Structured Patterns, and Practical Acceleration

Practical acceleration with DST depends critically on the sparsity pattern. Unstructured sparsity, while parameter-efficient, yields irregular memory access and limited GPU/TPU speedups. Structured DST variants overcome this:

• N:M and Constant Fan-in: SRigL and BDWP algorithms enforce fine-grained N:M constraints per row, enabling efficient kernel implementation and vectorized processing; e.g., 2:8 pattern delivers up to 1.75×–2× training and 13× inference speedups (Fang et al., 2023, Lasby et al., 2023).
• Diagonal and Block Patterns: DynaDiag maintains a fixed number of active diagonals, with CUDA BCSR implementation achieving up to 3.13× faster inference relative to unstructured baselines, while maintaining or very slightly reducing accuracy at extreme sparsities (Tyagi et al., 13 Jun 2025).
• Channel Granularity: Channel-aware DST (Chase) identifies sparse-amenable channels early in training and converts unstructured masks into contiguous channel removals, allowing 1.2–1.7× speedup on commodity GPUs without custom sparse kernels (Yin et al., 2023).
• Permutation-Augmented: PA-DST learns layerwise permutations in conjunction with a block or diagonal structured mask, recovering unstructured expressivity and delivering up to 2.9× faster inference while matching dense-level accuracy (Tyagi et al., 16 Oct 2025).
• Sparse Attention: Dynamic Top-K masking in transformer attention (e.g., SparseBalance, MTraining) leverages blockwise adjustability to balance compute across distributed systems, yielding up to 1.33×–6× end-to-end speedup for long-context language modeling (Xu et al., 15 Apr 2026, Li et al., 21 Oct 2025).

Empirical evaluations confirm that for diverse models (ResNet-50, ViT, GPT-2, BERT), structured DST achieves dense-comparable accuracy at 80–97% sparsity, with observed speedups tracking theoretical FLOP reductions when structure matches accelerator native layout.

5. Domain-Specific Applications and Extensions

Dynamic sparse training patterns have demonstrated efficacy in a variety of neural architectures and settings:

• Recurrent Neural Networks: Selfish-RNN extends DST with gate-wise sparsity redistribution and a sparse-averaged optimizer (SNT-ASGD), achieving superior perplexity over dense baselines and other sparse-to-sparse methods, especially when random regrowth is favored (Liu et al., 2021).
• Spatio-Temporal Forecasting: DynST generalizes DST to input-side sparsity, learning binary masks over sensors or image patches that filter low-importance regions while maintaining forecast accuracy, with up to 2× inference speedup (Wu et al., 2024).
• Extreme Multi-label Classification: DST with fixed fan-in, semi-structured sparse format, and auxiliary gradient-flow regularization (e.g., Spartex) enables XMC tasks with millions of labels to be trained in commodity GPU memory with only small accuracy tradeoffs (Ullah et al., 2024).
• Federated Learning: FedDST applies dynamic mask evolution per client in a federated setting, reducing both computation and communication by up to 5× while increasing accuracy under tight data budgets, exploiting the in-time self-ensembling effect and local adaptation (Bibikar et al., 2021).
• DRL/MARL: DST has been adapted to deep (multi-agent) RL, where dynamic mask evolution in value functions and policies tracks the nonstationary data distribution, reducing FLOPs by 10×–20× with negligible reward degradation (Sokar et al., 2021, Hu et al., 2024).

DST has also been extended to resource-constrained settings, where data-level DST (masking sensor inputs), or ultra-sparse distributed training (dynamic sparse attention) are essential for tractable large-scale training.

6. Empirical Patterns, Best Practices, and Limitations

Empirical analysis of dynamic sparse training reveals characteristic patterns and practical recommendations:

• Early in training, masks undergo rapid topological changes (exploration), before “settling” to stable, high-magnitude substructures that specialize by layer or channel (Yin et al., 2023, Hu et al., 2024).
• The dynamic nature of mask evolution exposes the model to a wide diversity of sparse subnetworks ("cheap tickets"), which, when superposed (e.g., Sup-tickets), can aggregate complementary features and further improve generalization, sometimes surpassing dense baselines (Yin et al., 2022).
• Hyperparameter choices crucially control performance: e.g., mask update interval, prune/growth fraction, trade-off between exploration and exploitation (DST-EE), or window sizes in sparse attention (Huang et al., 2022, Xu et al., 15 Apr 2026).
• Structured DST requires careful matching of block/channel/diagonal size with accelerator capabilities to realize hardware acceleration (Tyagi et al., 13 Jun 2025, Lasby et al., 2023, Fang et al., 2023).
• Extremely high sparsity (>95%) can induce accuracy degradation unless regularized by mask restructuring (ablation, redistribution) or adaptive thresholding (Lasby et al., 2023, Tyagi et al., 13 Jun 2025).

A limitation is that unstructured DST, despite parameter and FLOP savings, seldom yields proportional runtime gains without structured adaptation. Additionally, the dynamic mask schedule may induce additional implementation complexity, and convergence proofs for all regimes—especially in federated or nonstationary RL settings—remain limited.

7. Theoretical Guarantees and Ongoing Research Directions

Recent work provides theoretical underpinnings for DST:

• Sublinear convergence of the average gradient norm for the discrete-continuous optimization under mild assumptions (Huang et al., 2022).
• Multilevel mirror-descent approaches can linearly converge under Bregman-PL assumptions, reducing FLOPs to as little as 6% of dense SGD while preserving test accuracy (Lunk et al., 3 Feb 2026).
• DST is empirically shown to rank subnetworks by layer redundancy, thereby directly informing future compact architecture design (Liu et al., 2020).
• Extensions such as permutation-augmented patterns, meta-optimization of channel/block-level budgets, sparse evolutionary masking, and distributed context-aware attention are active areas of research aimed at closing accuracy/efficiency gaps across architectures and domains (Tyagi et al., 16 Oct 2025, Yin et al., 2023, Li et al., 21 Oct 2025, Xu et al., 15 Apr 2026).

Future extensions include hardware-software co-design for more general sparse kernels, adaptive or meta-learned masking strategies, and integration with federated, continual, and resource-adaptive learning regimes.

---

References:

• (Liu et al., 2021) Selfish Sparse RNN Training
• (Hu et al., 2024) Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training
• (Yin et al., 2023) Dynamic Sparsity Is Channel-Level Sparsity Learner
• (Wu et al., 2024) DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecasting
• (Ullah et al., 2024) Navigating Extremes: Dynamic Sparsity in Large Output Spaces
• (Xu et al., 15 Apr 2026) SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
• (Liu et al., 2020) Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• (Lasby et al., 2023) Dynamic Sparse Training with Structured Sparsity
• (Tyagi et al., 16 Oct 2025) Efficient Dynamic Structured Sparse Training with Learned Shuffles
• (Tyagi et al., 13 Jun 2025) Dynamic Sparse Training of Diagonally Sparse Networks
• (Huang et al., 2022) Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off
• (Liu et al., 2018) Dynamic Sparse Graph for Efficient Deep Learning
• (Sokar et al., 2021) Dynamic Sparse Training for Deep Reinforcement Learning
• (Gong et al., 2019) SparseTrain: Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
• (Li et al., 21 Oct 2025) MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
• (Bibikar et al., 2021) Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better
• (Yin et al., 2022) Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training
• (Lunk et al., 3 Feb 2026) Sparse Training of Neural Networks based on Multilevel Mirror Descent
• (Dietrich et al., 2021) Towards Structured Dynamic Sparse Pre-Training of BERT
• (Fang et al., 2023) Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design