Pretrain Ensembles: Methods & Insights

• Pretrain ensembles are methods that use diverse pretraining elements (seeds, data splits, architectures) to form robust model ensembles.
• They employ techniques such as checkpoint ensembling, subnetwork pruning, and dynamic heads to enhance efficiency and generalization.
• Empirical studies show these methods can boost accuracy by up to 1.6% and improve robustness against distribution shifts in low-data scenarios.

Pretrain ensembles refer to methodologies that leverage diversity originating from different stages, seeds, or forms of pretraining to construct ensembles of neural models. Unlike traditional ensembles that rely solely on independent random initializations or hyperparameter sweeps, pretrain ensemble approaches exploit diversity in pretraining (e.g., pretraining data, seeds, or modalities) or are explicitly structured to obtain ensemble-quality generalization with reduced training and inference costs. These methods play a pivotal role in modern transfer learning, model alignment, robust prediction, and efficient deployment.

1. Forms of Diversity in Pretrain Ensembles

Diversity is fundamental to the efficacy of any ensemble. In pretrain ensembles, diversity arises from several orthogonal sources:

• Pretraining Initialization: Models initialized from different random seeds during pretraining traverse distinct regions of the parameter space, inducing functional diversity even after subsequent finetuning (Eisenstein et al., 2023, Mustafa et al., 2020).
• Data Partitioning in Pretraining: Training on different subsets of pretraining data (“experts” versus “generalists” or different modalities) introduces representational divergence (Mustafa et al., 2020).
• Architectural Variations: Ensembles may incorporate diverse architectures, such as varying backbone depths, widths, or attention mechanisms, to further increase coverage of the hypothesis space (Mustafa et al., 2020).
• Structured Networks or Subnetworks: Approaches such as Multi-Ticket Ensembles extract subnetworks from the same pretrained model via pruning and regularization to maximize diversity without the need for separate pretraining runs (Kobayashi et al., 2022).
• Dynamic Heads and Shared Backbones: Model-agnostic dynamics where sparse, independently trained heads attach to a common backbone, as in NeuroTrails, induce controlled disagreement while amortizing feature extraction (Grooten et al., 23 May 2025).

Diversity from pretraining is often more robust to overfitting and distribution shift than diversity created by only downstream (finetuning) perturbations (Mustafa et al., 2020, Eisenstein et al., 2023).

2. Methodologies for Constructing Pretrain Ensembles

A spectrum of methodologies has been developed to construct ensembles leveraging pretraining:

• Selection from Large Pools of Pretrained Models: Algorithms select an ensemble from thousands of pretrained candidates using transferability proxies like kNN accuracy, followed by targeted finetuning and greedy assembly to minimize downstream loss (Mustafa et al., 2020). This is particularly effective in low-data transfer learning but generalizes to larger-scale domains.
• Ensembling via Pruning and Subnetwork Selection: Strategies such as Diverse Lottery Tickets identify “winning” sparse subnetworks within a single pretrained model; by enforcing regularization or sampling random masks, subnetworks that occupy different function basins are selected and ensembled (Kobayashi et al., 2022).
• Checkpoint Ensembles: Instead of independently pretrained networks, sequential snapshots from a single training run—captured at various checkpoints—are ensembled. This exploits diversity emerging during optimization as the model explores multiple minima in the local loss landscape (Chen et al., 2017).
• MotherNets and Function-Preserving Expansions: A minimal shared “MotherNet” is pretrained, then expanded into full ensemble members using function-preserving transformations and fine-tuning, balancing training cost and diversity via architectural clustering (Wasay et al., 2018).
• Weight Parameter Resampling: Estimate the empirical mean and variance of parameters in a fine-tuning phase via on-the-fly algorithms (e.g., Welford’s), and either set parameters to the mean (mean-resampled model) or sample multiple parameter vectors to form an ensemble (Liu et al., 2018).
• Dynamic Sparse Heads: NeuroTrails partitions a model into a shared backbone and dynamically trained, sparse heads; evolutionary strategies guide the sparsity patterns, ensuring each head traverses a distinct “neural trail” (Grooten et al., 23 May 2025).
• Reward Model Pretrain Ensembles: In alignment, ensemble members are constructed from reward models with different pretraining seeds, reducing shared spurious correlations and limiting reward hacking when compared to ensembles that differ only in fine-tuning seeds (Eisenstein et al., 2023).

The following table summarizes select representative methods:

Method	Ensemble Diversity Source	Key Efficiency Feature
Pretrained Pool Selection (Mustafa et al., 2020)	Pretraining data/seeds	Selects/fine-tunes top candidates
Diverse Lottery Tickets (Kobayashi et al., 2022)	Pruned subnetworks in same model	No extra pretrain; only pruning masks
Checkpoint Ensembles (Chen et al., 2017)	Dynamics during a single run	No extra training, only checkpointing
NeuroTrails (Grooten et al., 23 May 2025)	Sparse heads on shared backbone	Dynamic sparsity; amortized computation
Reward Model Ensembles (Eisenstein et al., 2023)	Pretraining seed diversity	Shared downstream data, diverse RMs

3. Theoretical Foundations and Mathematical Formulation

Pretrain ensemble methods rely on established mathematical formulations:

• Bayesian Posterior Integration: SG-MCMC approaches approximate the Bayesian posterior by sampling parameter vectors $\theta^{(m)}$ and constructing the predictive distribution as:

$$P(\tilde{y}\mid\tilde{x}, D) \approx \frac{1}{M} \sum_{m=1}^M P(\tilde{y}\mid\tilde{x}, \theta^{(m)})$$

where the diversity arises naturally from posterior sampling over different pretraining (and SGD) trajectories (Zhang et al., 2018).

• Parameter Resampling: Parameter distribution is estimated by tracking the running mean $\mu$ and variance $\sigma^2$ during a short fine-tuning stage:

$$\mu_{t+1} = \mu_t + \frac{\theta_{t+1} - \mu_t}{t+1}, \quad \sigma^2_{t+1} = \sigma^2_t + (\theta_{t+1} - \mu_t)(\theta_{t+1} - \mu_{t+1})$$

Ensembles are then constructed by resampling or adopting the mean parameterization (Liu et al., 2018).

• Aggregation in Reward Model Ensembles: Given reward models $r_m(x, y)$, the ensemble reward is $\overline{r}(x, y) = \mathrm{agg}(\{r_m(x, y)\}_m)$ using an aggregation function (mean, median, or mean minus standard deviation) (Eisenstein et al., 2023).
• Averaging Outputs vs. Weights: Prediction ensembling combines outputs:

$$M_{CE}(x_0) = \frac{1}{k} \sum_{i=1}^k M_{(i)}(x_0)$$

whereas in some smoother variants, parameters are averaged, with the caveat of potential parameter permutation misalignment (Chen et al., 2017).

4. Empirical Evaluation and Performance

Empirical studies consistently demonstrate that pretrain ensembles can achieve or surpass the accuracy and robustness of traditional ensembles at lower computational costs, with performance gains strongly tied to the nature of ensemble diversity:

• Low-Data Transfer Learning: Pretraining-driven model selection and ensembling attain state-of-the-art results across 19 Visual Task Adaptation Benchmark tasks, notably improving robustness to distribution shift, with expert selection outperforming downstream-only diversity methods by 1.2–1.6% in accuracy (Mustafa et al., 2020).
• Reward Model Alignment: Pretrain ensembles of reward models yield higher robustness to overoptimization and reward hacking, with win-rate improvements on policy alignment tasks at scale, though not fully eliminating spurious correlation exploitation (Eisenstein et al., 2023).
• Subnetwork Ensembles: Multi-Ticket Ensembles offer higher diversity as quantified by measures such as the Q-statistic, with ensemble gains (+1.5% on MRPC in GLUE) over dense and bagging baselines—but are sensitive to subnetwork quality (Kobayashi et al., 2022).
• Checkpoint Ensembles: On datasets such as CIFAR-10 and Reuters, checkpoint ensembling improves accuracy over minimum-validation-score selection by up to 0.0225 and leads to faster convergence (e.g., 50 epochs vs. 70 epochs) (Chen et al., 2017).
• Resource Efficiency: Paradigms such as NeuroTrails and MotherNets achieve dense-ensemble–comparable or superior accuracy with substantial reductions in FLOPs, parameters, and wall-clock time, validated on ResNet, Wide-ResNet, ImageNet, and LLaMA-350M (Grooten et al., 23 May 2025, Wasay et al., 2018).

5. Applications and Implications

Pretrain ensembles find utility across diverse domains and tasks:

• Low/Zero-Shot Generalization: In the low-sample regime, ensembles selected from pretrained pools provide performance surpassing large singular or non-ensemble models (Mustafa et al., 2020).
• Efficient Foundation Model Deployment: Methods like MotherNets and dynamic-head architectures enable rapid scaling of ensembles for foundation models across vision, language, and multimodal applications without linear cost increase (Wasay et al., 2018, Grooten et al., 23 May 2025).
• Robustness Against Distribution Shift: Pretrained ensemble diversity enhances resilience against common corruptions, adversarial inputs, or real-world domain shift, shown in robustness benchmarks such as ImageNet variants (Mustafa et al., 2020).
• Reward Model Robustness: In RLHF and alignment, using pretrain ensembles for reward modeling mitigates overoptimization and provides a more rigorous control against reward hacking, though systematic error in shared data remains (Eisenstein et al., 2023).
• Flexible Multimodal Systems: Frameworks such as AdaViT demonstrate that transformers pretrained on variable modality configurations enable robust transfer and ensembling across clinical imaging scenarios with heterogeneous input (Das et al., 4 Apr 2025).

6. Limitations, Challenges, and Future Directions

Several limitations and open directions are identified for pretrain ensemble approaches:

• Diversity–Quality Tradeoff: Simply maximizing diversity (e.g., via aggressive exploration outside the pre-train basin (Sadrtdinov et al., 2023) or random subnetworks (Kobayashi et al., 2022)) can degrade mean individual model quality, motivating algorithms such as StarSSE that decouple initial fine-tuning from diversity-seeking snapshots, retaining proximity to the pretrained origin.
• Underspecification and Shared Modes: In reward models, even ensembles created from distinct pretraining seeds can converge to shared biases when the downstream data is limited or non-representative, limiting their ability to fully suppress reward hacking (Eisenstein et al., 2023).
• Resource Constraints: While checkpoint and subnetwork ensembles are efficient, some architectures (e.g., large pools of pretrained models) still impose significant memory or inference overhead, motivating the pursuit of shared-backbone or dynamic-sparsity paradigms (Grooten et al., 23 May 2025).
• Distance-Aware Uncertainty: Standard ensembling provides only limited uncertainty estimates, especially when all members are close in function space. Distance- or distribution-awareness remains a challenge for both robust prediction and alignment tasks (Eisenstein et al., 2023).
• Generalization Beyond Vision: While general principles apply, some methods’ effectiveness is highly domain-specific (e.g., medical imaging variable modality handling (Das et al., 4 Apr 2025)); expanding these to general NLP or cross-modal ensembles remains ongoing work.
• Fine-grained Regularization: Further research is needed to develop regularization methods that balance diversity and individual model performance, both at structure (masking, pruning) and data (augmentation, class balancing) levels.

Promising directions include adaptive balance of diversity and transfer benefits (Sadrtdinov et al., 2023), more granular uncertainty quantification, and further integration of ensemble techniques with foundation model development and efficient, heterogeneous-data deployment.

7. Representative Pretrain Ensemble Techniques: A Comparative Table

Technique	Source of Diversity	Resource Efficiency	Robustness to Shift
StarSSE (Sadrtdinov et al., 2023)	Divergent snapshots post-fine-tune	High (single training trajectory)	High within pre-train basin
Pretrained Pool Selection (Mustafa et al., 2020)	Data/seed/architecture in pretrain	Moderate (selection/fine-tune only top)	High up/downstream
Multi-Ticket Ensemble (Kobayashi et al., 2022)	Pruned subnetworks, regularization	Very high (single pretrain)	Variable (depends on subnetwork strength)
Checkpoint Ensembles (Chen et al., 2017)	Time-dynamic checkpointing	Highest (single run)	Good for stochasticity
Reward Model Pretrain Ensembles (Eisenstein et al., 2023)	Pretrain seed, aggregation	High (no extra training)	Partial to systematic error

• (Chen et al., 2017): Checkpoint Ensembles: Ensemble Methods from a Single Training Process
• (Zhang et al., 2018): Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
• (Liu et al., 2018): Make (Nearly) Every Neural Network Better: Generating Neural Network Ensembles by Weight Parameter Resampling
• (Wasay et al., 2018): MotherNets: Rapid Deep Ensemble Learning
• (Mustafa et al., 2020): Deep Ensembles for Low-Data Transfer Learning
• (Kobayashi et al., 2022): Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
• (Sadrtdinov et al., 2023): To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
• (Eisenstein et al., 2023): Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
• (Das et al., 4 Apr 2025): AdaViT: Adaptive Vision Transformer for Flexible Pretrain and Finetune with Variable 3D Medical Image Modalities
• (Grooten et al., 23 May 2025): NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling

---

Pretrain ensembles exemplify a paradigm shift in ensemble learning, where diversity originates from the pretraining pipeline itself—whether by seed, data partition, architecture, or structured subnetwork extraction. These methods offer scalable accuracy gains, improved robustness, and resource-efficient alternatives to classical deep ensembles, and they continue to shape research in transfer learning, uncertainty quantification, foundation model deployment, and alignment.