Lottery Ticket Ensembling Methods

• Lottery Ticket Ensembling is a set of methods that extract sparse, high-performing subnetworks using pruning techniques to identify winning tickets.
• LTE employs strategies such as iterative magnitude pruning, weak mask aggregation, and dynamic stochastic ensembles to generate diverse subnetworks.
• LTE enhances predictive diversity, adversarial robustness, and computational efficiency, as demonstrated by improved performance in NLP and computer vision tasks.

Lottery Ticket Ensembling (LTE) refers to a family of methodologies that leverage the extraction and combination of sparse, high-performing subnetworks—termed "lottery tickets"—from neural networks to improve robustness, predictive diversity, and efficiency across numerous machine learning tasks. These approaches synthesize insights from the Lottery Ticket Hypothesis and ensemble learning, and span mechanisms at both the explicit architectural and implicit statistical levels.

1. Conceptual Foundations

The Lottery Ticket Hypothesis (LTH) asserts that within a dense, randomly initialized neural network, there exist sparse subnetworks which, when appropriately trained, achieve test accuracy comparable to the original model. LTE extends this hypothesis by not only seeking single "winning tickets" but systematically assembling multiple such subnetworks—either across independently pruned models, via in situ architectural decomposition, or through stochastic or randomized procedures—in order to aggregate their predictive strengths.

In broad terms, LTE encompasses:

• Explicit ensembling of independently obtained winning tickets, where masks are generated (typically via magnitude pruning or its variants) and subnetworks are trained or fine-tuned in isolation before aggregation (Kobayashi et al., 2022).
• In-network ensembling, conceptualizing each wide network as containing an intrinsic ensemble of specialized subnetworks that act as partially independent predictors; the aggregation occurs naturally via the output layer, yielding a variance reduction analogous to the Central Limit Theorem (Liu et al., 2023).
• Algorithmic ensembling in the pruning process, where ensembles of weak masks or noisy subnetworks are aggregated through mask operations (e.g., union or averaging) to yield high-quality, denoised winning tickets with reduced computational cost (Jaiswal et al., 2023).

LTE leverages the structural and predictive diversity emerging from the subnetwork extraction process, which is critical for both standard ensembling performance gains and specialized defenses such as adversarial robustness (Peng et al., 2022).

2. Subnetwork Extraction and Diversity Generation

The practical realization of LTE relies on several subnetwork extraction and mask-aggregation protocols:

• Iterative Magnitude Pruning (IMP): IMP cycles alternate between training (or fine-tuning) and removing a fraction of lowest-magnitude parameters from the network, followed by rewinding or reinitialization at each round. By varying random seeds, hyperparameters, or introducing regularizers (e.g., $L_1$ per-mask penalties), one can generate subnetwork families that are diverse in both structure and predictive behavior (Kobayashi et al., 2022).
• Weak Mask Aggregation: ISP replaces repeated full IMP-training-prune cycles with a "soup" of weak masks: from a pretrained or partially trained model, multiple masks are generated via short fine-tuning runs under varying data subsets and hyperparameters, then aggregated by union or similar operations to reduce mask-selection variance (Jaiswal et al., 2023).
• Architectural and Sparsity Diversity: Explicitly sampling subnetworks across different base architectures (e.g., ResNet variants, WideResNets) and a wide range of sparsity levels increases adversarial and functional diversity, a property measured through transferability metrics and shown to enhance ensemble performance (Peng et al., 2022).

Diversity among ensemble members is quantified through metrics such as average prediction disagreement, Q-statistics, error ratios, negative double fault, and subnetwork mask overlap. Empirically, subnetworks extracted with IMP and variants typically show only 90–99% overlap in mask elements (further reduced with regularizers), producing more complementary prediction errors compared to dense fine-tuning (Kobayashi et al., 2022).

3. Ensemble Construction and Prediction Aggregation

LTE methods aggregate subnetwork predictions to generate the final output. The aggregation strategy depends on the underlying ensemble construction:

• Averaging Logits/Probabilities: For $K$ extracted subnetworks $\{(\theta_s, m_s)\}_{s=1}^K$, the ensemble prediction is computed as

$$\hat{y} = \arg\max_{c}\left(\frac{1}{K}\sum_{s=1}^K f_c(\theta_s \odot m_s; x)\right),$$

or, equivalently, by averaging predicted class probabilities (Kobayashi et al., 2022).

• Randomized Ensemble Sampling: In adversarial defense frameworks, the ensemble configuration is re-sampled at each inference, randomly choosing architectures, sparsity levels, and subnetwork instances, thereby increasing attacker uncertainty (Peng et al., 2022).
• Weighting Schemes: Subnetworks may be weighted by calibration or validation accuracy, though simple uniform averaging is most commonly reported.

In the context of in-network ensembling, the output layer automatically computes a weighted sum over $O(N)$ “ticket” subnetworks, where $N$ is model width, leading to a collective reduction in output variance (Liu et al., 2023).

4. Theoretical Mechanisms and Scaling Laws

A key insight from LTE is the emergence of improved generalization and robustness through ensemble effects among structurally distinct subnetworks:

• Variance Reduction: The combined prediction error across $n$ approximately independent tickets decays as $O(1/n)$. When the number of effective tickets grows proportionally to network width $N$, this produces an $N^{-1}$ scaling of mean-squared error (MSE), contrasting with the $K$0 decay predicted by approximation theory in some settings (Liu et al., 2023).
• Central Limit Theorem Mechanism: As width increases, the distribution of output errors from the ensemble converges to a Gaussian, with the variance shrinking inversely with the number of independently aggregated tickets—a phenomenon observed directly in empirical loss histograms for both SiLU and ReLU activations (Liu et al., 2023).
• Transferability and Adversarial Diversity: Low adversarial transferability among tickets—quantified by $K$1, the robust accuracy of subnetwork $K$2 on adversarial examples from $K$3—implies greater defense effectiveness when assembling heterogeneous subnetwork libraries (Peng et al., 2022).

Table: Illustrative Metrics and Outcomes in Two LTE Regimes

Approach	Subnetwork Diversity Metric	Performance Improvement
Multi-Ticket Ensemble (BERT, (Kobayashi et al., 2022))	Pairwise prediction disagreement up by 10–20% (vs dense)	+1.52 (MRPC), +0.86 (STS-B) GLUE tasks
Adversarial LTE (ResNet/WRN, (Peng et al., 2022))	Cross-architecture transferability reduced; up to 9% higher robust accuracy in cross-arch attacks	+3–10% robust, +1% clean (vs dense); +15.4% robust (vs single)

This framework theoretically marries classical ensembling variance reduction with structural mechanisms made accessible by neural pruning and architectural breadth.

5. Practical Algorithms and Computational Trade-offs

LTE has motivated efficient algorithms addressing both computational bottlenecks and sensitivity to subnetwork extraction quality:

• Instant Soup Pruning (ISP): ISP achieves lottery-ticket-quality masks at the cost of a single IMP pass by aggregating multiple weak, noisy masks, each generated with small, randomized subsets of parameters, learning rates, or data. Theoretical error bounds suggest rapid decay in aggregation error with the number of independent masks. Empirical results show that ISP matches or outperforms full IMP in both vision (CLIP) and language (BERT) settings, at sparsity up to 90%, with only one full training cost (Jaiswal et al., 2023).
• Dynamic Stochastic Ensembles: For adversarial defense, dynamic stochastic resampling of the ensemble at every inference instance is recommended, leveraging diversified subnetwork libraries to hinder attack transferability. Libraries with ≥30–40 tickets covering ≥3–4 architectures are empirically optimal (Peng et al., 2022).
• Ensemble Size and Sensitivity: Performance gains typically saturate for ensemble sizes $K$4–$K$5; too aggressive pruning can degrade ticket quality and nullify ensemble benefits (Kobayashi et al., 2022).

Practitioners are advised to validate that pruned tickets maintain near-baseline accuracy, select regularizer configurations (e.g., random–lt masking) to maximize diversity, and perform ablation on ensemble size and sparsity thresholds for optimal trade-offs.

6. Applications and Empirical Outcomes

LTE achieves measurable improvements in several domains:

• Natural Language Processing: Multi-Ticket Ensemble applied to BERT-base increases performance on MRPC from 83.48% (single) to 85.05% (ensemble, random–lt), exceeding improvements from standard dense fine-tuning or bagging (Kobayashi et al., 2022).
• Computer Vision: On CIFAR-10 and several other benchmarks, ISP outperforms full IMP at various sparsities, demonstrating both higher accuracy and reduced computational requirements. On CLIP-ViT-B32, ISP with 70% sparsity achieves higher test accuracy across diverse vision tasks versus classic IMP (Jaiswal et al., 2023).
• Adversarial Training: Dynamic stochastic ensembles of robust lottery tickets achieve clean accuracy of 87.0% and robust accuracy of 67.7%, +15.4% over single-model adversarial training (Peng et al., 2022).
• Scaling Laws: LTE provides a mechanistic basis for $K$6 loss scaling in wide networks, with implications for interpreting empirical neural scaling law results observed in LLMs (Liu et al., 2023).

7. Limitations, Open Questions, and Theoretical Connections

Several limitations and future directions are recognized:

• Subnetwork Quality Dependence: LTE’s gains require that extracted tickets individually maintain high test accuracy. On certain tasks (e.g., CoLA, QNLI), standard pruning methods may fail, highlighting a need for more advanced or structured pruning techniques (Kobayashi et al., 2022).
• Diversity Metrics vs. Performance: Existing diversity measures (mask overlap, disagreement) are only loosely correlated with final ensemble gains; improved theoretical proxies are needed (Kobayashi et al., 2022).
• Orthogonality and Independence Assumptions: Scaling-law derivations presume tickets are approximately unbiased and orthogonal, but in practice, effective independence may be reduced by correlation among subnetwork residuals (Liu et al., 2023).
• Extension to New Domains: Current LTE formulations are empirically validated for classification, regression, and adversarial robustness. Extending LTE-oriented mask aggregation and ensembling to generative tasks (translation, summarization) and to larger model families remains ongoing.
• Theoretical Integration: LTE bridges constructive, subnetwork-based views with mean-field and statistical-physics approaches describing finite- and infinite-width neural networks. A plausible implication is that scaling law regularities in modern LLMs may originate from massive in situ lottery ticket ensembling (Liu et al., 2023).

LTE represents a rapidly evolving intersection of pruning theory, ensemble learning, robustness, and neural scaling phenomena, offering both practical efficiency and a deeper theoretical understanding of deep networks' predictive organization.