Golden Ticket Hypothesis Overview

• Golden Ticket Hypothesis is a theory that sparse subnetworks, or winning tickets, within large neural networks can be retrained to achieve comparable or superior accuracy to the original model.
• The hypothesis is validated using iterative magnitude pruning with weight rewinding, where low-magnitude weights are systematically removed to reveal effective subnetworks.
• Empirical studies show that these winning tickets not only improve training efficiency but also transfer across diverse architectures and tasks while maintaining robust performance.

The Golden Ticket Hypothesis, commonly known as the Lottery Ticket Hypothesis (LTH), posits that within large overparameterized neural networks, there exist sparse subnetworks—termed "winning tickets"—that, when initialized identically to the parent network and trained in isolation, match or sometimes exceed the performance of the original dense model. This paradigm has become a central object of study in deep learning, touching on foundational questions of overparameterization, trainability, generalization, efficiency, and transferability across diverse architectures and tasks.

1. Formal Statement and Definitional Framework

The original formalization (Liang et al., 2021) considers a neural network $f(x; \theta)$ with parameters $\theta \in \mathbb{R}^P$ and a binary mask $m \in \{0,1\}^P$. The subnetwork is defined by $f(x; \theta \odot m)$, where $\odot$ denotes element-wise product. The hypothesis asserts that there exists a sparse mask $m^\star$ with $\|m^\star\|_0 \ll P$ such that, with the network initialized at $\theta_0$, the following holds:

• The pruned network trained from $\theta_0 \odot m^\star$ matches or exceeds the original model's performance:

$$\mathcal{L}_\text{val}(f(\cdot; \theta_0 \odot m^\star)) \leq \mathcal{L}_\text{val}(f(\cdot; \theta_0)) + \epsilon$$

• It outperforms random subnetworks of the same size:

$$\mathcal{L}_\text{val}(f(\cdot; \theta_0 \odot m^\star)) < \mathcal{L}_\text{val}(f(\cdot; \theta_0 \odot m_\text{rand}))$$

A "winning ticket" thus refers to such a mask and corresponding initialization; the term "golden ticket" is used interchangeably.

Rigorous criteria for a true "winning ticket" require, as codified in (Ma et al., 2021):

• The ticket matches the accuracy of a well-trained dense baseline,
• Outperforms both a random re-initialization and a matching small-dense network,
• Holds under a "non-trivial" sparsity ratio and a sufficiently strong training regime (learning rate and epochs).

2. Iterative Magnitude Pruning and Algorithmic Realizations

The canonical methodology for uncovering winning tickets is Iterative Magnitude Pruning (IMP) (Paul et al., 2022, Maene et al., 2021). The IMP-WR (with weight rewinding) process is:

1. Train the full dense model to a predefined point (often early epochs for "rewinding").
2. Prune (globally or layerwise) a fraction of weights with the lowest magnitudes, yielding mask $m$.
3. Reset the unpruned weights to their values at initialization or the rewind point.
4. Repeat the train–prune–rewind cycle iteratively, updating $m$ each time.
5. The final mask after $L$ rounds defines the sparse winning ticket.

Variants consider different pruning criteria (global magnitude, layerwise importance (Vandersmissen et al., 2023)), rewind schemes, and structured pruning (e.g., attention heads in Transformers (Liang et al., 2021, Prasanna et al., 2020)).

3. Theoretical Foundations and Geometric Mechanisms

Provable existence results for lottery tickets extend beyond fully-connected ReLU nets to CNNs and ResNets, with broad classes of activation functions (Burkholz, 2022). The proofs revolve around subset-sum arguments and covering numbers: a sufficiently wide random network can, with high probability, be pruned to approximate any fixed target network of similar depth and dimension to arbitrary accuracy.

Recent work locates the geometric core of LTH in the error landscape:

• Winning ticket masks specify axial subspaces that intersect linearly connected basins of low loss (Paul et al., 2022).
• SGD exhibits strong robustness to perturbations near these axes, enabling retraining to return to the same minima after masking.
• The maximal single-step pruning ratio is dictated by the local Hessian spectrum (flatness): flatter directions (small eigenvalues) allow more aggressive pruning without loss of connectivity.

In IMP, retraining after each prune is essential to re-equilibrate the weight distribution, allowing further pruning cycles to exploit newly created "flat" directions.

4. Empirical Regimes, Criteria, and Extensions

Extensive empirical work has clarified the operational boundary of the Golden Ticket Hypothesis (Ma et al., 2021, Liu et al., 2021):

• The existence of winning tickets depends on the interplay of learning rate, number of epochs, architecture (with or without residuals), and degree of overparameterization relative to the data regime.
• At large learning rates or high-capacity models and large datasets, the benefit of original-initialization vanishes; prune-retrain and random-reinit perform equally.
• For modern models, prune-and-fine-tune—using trained weights rather than resetting to initialization—almost always outperforms the classic sparse re-training at high sparsities.

Extensions to pre-trained models, including supervised and self-supervised regimes, demonstrate that highly sparse tickets maintain full transfer performance for downstream classification, detection, and segmentation (Chen et al., 2020). In graph neural networks, simultaneous pruning of adjacency and parameter matrices is feasible, but excessive sparsity in the graph structure degrades accuracy sharply (Hui et al., 2023).

Generalizations further reveal that winning tickets are not unique: multiple structurally distinct subnetworks can serve as effective tickets, with only a small stable core shared among them (Vandersmissen et al., 2023).

5. Transferability, Universality, and Domain-Specific Findings

Tickets discovered on large, diverse datasets can act as universal initializations for a wide array of tasks and optimizers (Morcos et al., 2019, Burkholz et al., 2021). Theoretically, depth-dependent sparse constructions can yield a single "universal ticket" that approximates any function in a wide family, requiring only retraining of a final linear readout (Burkholz et al., 2021).

Empirical transfer experiments demonstrate near-oracle accuracy for tickets transferred across datasets (e.g., ImageNet→CIFAR) or even between neural ODE solvers with appropriate adjustment for dynamical system "universality classes" (Prideaux-Ghee, 2023). In GNNs, graph lottery tickets (GLTs) can be transferred across graphs, preserving accuracy if crafted with robust loss forms and min–max adversarial strategies during pruning (Hui et al., 2023).

6. Novel Phenomena, Limitations, and Open Directions

Not all architectures or training setups yield easily identifiable golden tickets. Key caveats and findings include:

• The phenomenon is closely tied to stable optimization: high SGD noise or insufficient training epochs obscure the winning ticket effect (Maene et al., 2021).
• Weight resetting in standard LTH "forgets" previous learning; recursive strategies that combine structural growth with pruning preserve and extend the functional core, mimicking biological "juvenile states" of learning (Zhang, 2021).
• In pre-trained models (huge transformers, large vision backbones), many random subnetworks are trainable provided sufficient size—indicative of an optimization landscape rich in equally-good "lottery tickets" rather than a single rare subnet (Prasanna et al., 2020).
• Layerwise and structured importance metrics extend the classical global-magnitude approach, offering more robustness and revealing non-uniqueness of tickets (Vandersmissen et al., 2023).
• Provable constructions show logarithmic overparameterization suffices for LTH in deep CNNs/ResNets, with minimal assumptions on activation functions (Burkholz, 2022).

Active research includes practical methods for faster ticket discovery (e.g., LOFT filter-wise distributed pretraining (Wang et al., 2022)), robust design of transferable tickets, understanding "super ticket" regimes where pruning improves generalization, and formal links to renormalization group theory in scientific ML contexts (Prideaux-Ghee, 2023).

7. Representative Algorithmic and Mathematical Summary

Step	Procedure	Purpose
Train	Full model trained to convergence or early rewind point	Establish baseline and initial weight distribution
Prune	Remove a fraction (global/layerwise/structured) of weights by criterion	Induce sparsity, define mask $m$
Rewind/Reset	Restore remaining weights to $\theta_0$ or chosen rewind state	Test "initialization lottery" property
Retrain	Retrain the sparse network in isolation	Evaluate ticket accuracy and convergence
Iterate	Repeat prune–rewind–retrain as necessary	Drive to target sparsity while preserving or improving acc.

Mathematical definition of a winning ticket (as per (Ma et al., 2021)):

$$\begin{align*} A_{\text{LT}} &\approx A_{\text{PRE}}, \ A_{\text{LT}} &> A_{\text{RR}}, \ A_{\text{LT}} &> A_{\text{SD}} \end{align*}$$

where $A_{\text{PRE}}$ is the accuracy of the well-trained dense baseline, $A_{\text{LT}}$ that of the ticket, $A_{\text{RR}}$ for random reinit, and $A_{\text{SD}}$ for a small-dense network of matching parameter count.

• (Liang et al., 2021) Super Tickets in Pre-Trained LLMs
• (Paul et al., 2022) Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?
• (Ma et al., 2021) Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• (Vandersmissen et al., 2023) Considering Layerwise Importance in the Lottery Ticket Hypothesis
• (Maene et al., 2021) Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
• (Burkholz, 2022) Convolutional and Residual Networks Provably Contain Lottery Tickets
• (Morcos et al., 2019) One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• (Hui et al., 2023) Rethinking Graph Lottery Tickets: Graph Sparsity Matters
• (Chen et al., 2020) The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• (Wang et al., 2022) LOFT: Finding Lottery Tickets through Filter-wise Training
• (Zhang, 2021) Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches
• (Burkholz et al., 2021) On the Existence of Universal Lottery Tickets
• (Prideaux-Ghee, 2023) Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers
• (Kölle et al., 14 Sep 2025) Investigating the Lottery Ticket Hypothesis for Variational Quantum Circuits

This literature collectively establishes the Golden Ticket Hypothesis as a robust, mathematically sound, and widely-generalizable property of modern neural architectures, with ramifications for efficient training, model compression, transfer learning, and theoretical understanding of deep learning landscapes.