Dynamical Lottery Ticket Hypothesis

• The Dynamical Lottery Ticket Hypothesis is a theory explaining how training dynamics and weight updates reveal effective sparse subnetworks (winning tickets).
• It extends the original Lottery Ticket Hypothesis by incorporating data-dependence, gradient-based pruning, and transfer learning to enhance model efficiency.
• This framework informs innovative pruning methods and structured sparsity designs that improve both computational performance and hardware efficiency.

The Dynamical Lottery Ticket Hypothesis builds upon the original Lottery Ticket Hypothesis by emphasizing the centrality of training dynamics—in particular, the trajectory of weight updates, initialization-dependent processes, and the influence of stochastic elements—in the emergence and effectiveness of sparse trainable subnetworks (“winning tickets”) in deep neural networks. While the foundational hypothesis posited that favorable weight initializations seed subnetworks amenable to effective isolated training, extensions and subsequent studies have refined this perspective to include effects of data, transferability, multiple solutions, structured pruning, and theoretical constructs linking winning tickets to broader function approximators and learnable representations.

1. Evolution from Static to Dynamical Perspective

The Lottery Ticket Hypothesis (LTH) originally identified “winning tickets” as sparse masks $m \in \{0,1\}^d$ over weights $\theta$ in an overparameterized network $f(x; \theta)$, such that $f(x; m \odot \theta_0)$, when trained in isolation from the same random initialization $\theta_0$, achieves comparable accuracy to the full dense network. However, empirical and theoretical analyses have shown that the static initial mask is not the sole determinant of success. Instead, the emergence and trainability of winning tickets depend critically on the sequence of weight updates and the state of the network at specific points in its optimization trajectory.

In the transfer learning extension, a key mechanism is “late resetting,” in which the mask is applied to parameters $\theta_S$ obtained from training on a source task, yielding $m \odot \theta_S$ rather than $m \odot \theta_0$ for transfer to a target task. This refinement demonstrates that the history of training directly influences the efficacy and transferability of sparse subnetworks (Mehta, 2019). Formally, the Ticket Transfer Hypothesis posits the existence of $m$ and $\theta^* \in \mathcal{P}_S$ (the source task training trajectory) such that the pruned-and-reset subnetwork matches or exceeds both accuracy and speed of the dense model on the target.

The dynamical viewpoint is further substantiated by findings that, in transfer scenarios and under broader pruning and retraining regimes, performance depends not only on the existence of suitable masks or “lucky initializations,” but also on how weight values evolve—supporting the interpretation that the winning ticket is a product of both favorable initialization and the sequence of learning dynamics.

2. Multiplicity and Distribution of Winning Tickets

Experimental work demonstrates that winning tickets are not unique; instead, there is a distribution over high-performing sparse subnetworks capable of reaching similar test accuracy, even from a fixed initialization (Grosse et al., 2020). For a fixed set of initial weights, stochasticity in training (e.g., data shuffling, mini-batch order, and noise in gradients) leads to the selection of statistically distinct winning tickets. The overlap between masks of such subnetworks aligns closely with hyper-geometric predictions, indicating that different tickets are almost independent, except when randomness during training is reduced, in which case overlap increases.

This multiplicity suggests that LTH does not expose a single hidden subnetwork but instead that the training process, influenced by both initialization and ongoing stochastic dynamics, dynamically selects specific winning subnetworks from a large ensemble. The identification and characterization of the distribution of winning tickets opens a path toward probabilistic understanding and potentially new algorithms that exploit structural properties of the winning ticket set.

3. Data-dependence and Gradient-based Pruning

A significant evolution in the methodology for identifying winning tickets involves incorporating data-dependent criteria into the pruning process (Lévai et al., 2020). The SNIP method, for instance, evaluates the saliency of each parameter not only by its absolute value but by the product $|w_i \cdot g_i|$, where $g_i$ is the average absolute gradient of the loss with respect to the weight over the training set:

$$g_i = \frac{1}{n} \sum_{j=1}^n \left| \frac{\partial \mathcal{L}(x_j, w_i)}{\partial w_i} \right|$$

Selecting weights based on this criterion ensures that those with high functional impact on the loss (i.e., those that are highly “dynamic” and responsive to the data) are retained, reinforcing the dynamical characterization of winning tickets. Experimental results show that gradient-aware methods yield higher-quality subnetworks under both initialization- and training-based pruning, as compared to magnitude pruning alone.

This approach further strengthens the interpretation that the success of a winning ticket depends on both its ability to respond to the training signal and its plasticity as mediated by the data and loss landscape throughout learning. Notably, it bridges the LTH with core ideas in optimization theory concerning sensitivity and the dynamics of stochastic gradient descent.

4. Practical Manifestations: Transfer Learning, Efficiency, and Tweaking

Applied work underscores the role of dynamics in not only locating but also “spending” or retraining winning tickets. Architecture and training recipe modifications—such as replacing ReLU with smooth activations, adding skip connections, introducing layerwise re-scaled initializations, and employing label smoothing or knowledge distillation—have been shown to improve the performance of winning tickets, especially at high sparsity levels (Jaiswal et al., 2021). These “tweaks” produce smoother loss landscapes, facilitate stable convergence, and can significantly boost the accuracy of sparse subnetworks over standard retraining regimes.

In the transfer learning context, the late-reset strategy ($m \odot \theta_S$) demonstrates that not just the initial subnetwork structure but the path through the weight space taken during source task training determines whether a subnetwork will transfer successfully. Pruning strategies that explicitly account for the evolving relationships between initialization, data, and representations offer a route to effective, efficient, and transferable sparse model design.

Furthermore, the efficiency of ticket identification and retraining is boosted by strategies such as training on carefully selected Pruning-Aware Critical sets (PrAC sets)—subsets of the data defined by high forgetting or pruning-induced prediction differences—substantially reducing the amount of data and computation required to find effective winning tickets (Zhang et al., 2021). This co-design approach integrates data selection with model sparsification as a coupled dynamic process.

5. Theoretical Foundations and Implications

The theoretical underpinnings of the dynamical lottery ticket phenomenon have been developed across multiple axes:

• In PAC-Bayesian terms, the generalization ability of winning tickets is linked to the flatness or sharpness of the loss minimum they converge to, and the distance traveled from initialization. Winning ticket subnetworks tend to occupy sharper minima and are more robust if their learned weights remain close to initialization, as confirmed using spike-and-slab priors in PAC-Bayes bounds (Sakamoto et al., 2022).
• Notions from dynamical systems have been employed to rationalize the structure and transferability of winning tickets; for instance, the connection between residual block stacking and Euler discretizations of ODEs in deep networks, and the interpretability of sparse block structures as “time-step refinements” (Chen et al., 2021).
• The existence of universal (task-agnostic) lottery tickets has been rigorously proved for sufficiently overparameterized networks, relying on technical innovations such as depth-dependent subset sum results and layer-wise error control (Burkholz et al., 2021). These constructions permit a subnetwork to serve as a universal approximator for an entire function family, requiring only a tailored output readout, and support adaptive reuse in dynamically changing environments.
• The distributional and stochastic nature of the winning ticket solution set suggests a nuanced probabilistic structure rather than a singular optimal subnetwork, motivating new perspectives for both theory and generative pruning heuristics (Grosse et al., 2020).

6. Structured and Hardware-Efficient Lottery Tickets

Early LTH results focused on unstructured sparsity, which is unfavorable for efficient hardware acceleration. Recent work demonstrates that winning tickets with structured sparsity—such as channel-wise or group-wise masks—can be obtained via post-processing steps following standard iterative magnitude pruning (Chen et al., 2022). Approaches that refill pruned weights within critical channels, or regroup weights into dense blocks optimized for GEMM operations, achieve parity or superior performance with dramatic reductions in inference time, meeting practical deployment constraints. These findings highlight a further dynamical aspect: the iterative and potentially post-hoc refinement of sparse structures to exploit both trainability and computational efficiency, aligning with the broader aim of efficient, deployable sparse learning.

7. Controversies, Guidelines, and the Path Forward

The existence and utility of winning tickets are now understood to be tightly coupled to network architecture, optimization regime, and training hyperparameters. Notably, rigorous definitions and criteria for what constitutes a “Jackpot” winning ticket, as opposed to weaker variants, clarify prior inconsistencies and underscore that such subnetworks may only be reliably identified under certain conditions (notably, low learning rates, presence of residual connections, ample overparameterization, or insufficient training) (Ma et al., 2021).

Empirical guidelines have emerged for practitioners, addressing optimal learning rate selection, pruning methods (iterative vs. one-shot), architecture type, and sparsity levels. The literature converges on the view that the lottery ticket phenomenon is best understood as a consequence of the interplay between initialization, dynamic training trajectories, data sensitivity, and the stochastic selection among many candidate subnetworks—rather than the uncovering of a singular, static mask hardwired into the network at initialization.

A plausible implication is that future research may benefit from algorithmic innovations that explicitly model and manipulate the dynamic selection pressures of training—not only by mask optimization but also by adaptive data selection, continuous rewinding, architecture design (e.g., via structured blocks), and even through theoretical approaches linking sparse trainability to broader properties of function approximation and generalization in non-convex landscapes.