Snapshot Ensembles in Deep Learning

• Snapshot ensembles are an ensemble method that trains one network with cyclic learning rates, capturing multiple local minima as distinct checkpoints.
• They improve performance by averaging predictions from diverse snapshots, achieving lower error rates on benchmarks like CIFAR-10/100 and ImageNet.
• Advanced variants extend the approach for transfer learning, clustering, and robustness, while addressing practical trade-offs in inference efficiency.

Snapshot ensembles are ensemble methods that extract multiple predictors from a single training trajectory rather than training multiple models independently. In the neural-network formulation introduced by Huang et al., one network is trained with a cyclic learning-rate schedule so that optimization repeatedly converges to several local minima; the resulting checkpoints are saved as “snapshots” and their predictions are averaged at test time (Huang et al., 2017). Subsequent work has treated the same basic pattern—one trajectory, multiple competent but non-identical models—as a reusable design principle, extending it to transfer learning, robustness, uncertainty quantification, consistency under periodic retraining, knowledge-graph link prediction, deep clustering, and inductive logic programming (Sadrtdinov et al., 2023, Shaban et al., 2024, Piróg et al., 2023, Liu et al., 28 Oct 2025).

1. Core formulation

The original method addresses a familiar tension in deep learning: ensembles are typically more robust and accurate than single networks, but a conventional ensemble of $M$ models usually costs about $M$ times as much training compute. Snapshot Ensembling seeks the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost by training a single neural network, converging to several local minima along its optimization path, and saving the model parameters (Huang et al., 2017).

Its mechanism is a cyclic learning-rate schedule, with $M$ cycles over a fixed training budget $T$. The generic form is

$$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$

where $t$ is the iteration number, $T$ is the total number of training iterations, $M$ is the number of learning-rate cycles, $\lceil T/M \rceil$ is the cycle length in iterations, and $f$ is a monotonically decreasing function over a cycle. In the reported experiments, $M$0 is instantiated as a shifted cosine:

$M$1

with $M$2 the restart learning rate (Huang et al., 2017).

Within each cycle, the learning rate starts high and anneals toward $M$3, encouraging convergence by the cycle end. At the end of the cycle—just before the learning rate is reset upward—the current parameter vector is saved as a snapshot. The next restart perturbs optimization enough to leave the current basin and move toward a different region of parameter space. Learning rate is updated per iteration rather than per epoch, so relatively short cycles can still produce useful convergences (Huang et al., 2017).

At inference time, the snapshots form an explicit ensemble: the models do not share weights at test time, and their softmax outputs are averaged. The original paper specifically averages the last $M$4 snapshots, because later snapshots generally have lower individual test error due to receiving more total optimization (Huang et al., 2017).

2. Empirical profile in supervised deep learning

The original study evaluates ResNet-110, Wide-ResNet-32, DenseNet-40, DenseNet-100, and ResNet-50 on CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and ImageNet, and reports that Snapshot Ensembles consistently improve over strong single-model baselines under the same training budget (Huang et al., 2017). The abstract highlights DenseNet Snapshot Ensemble error rates of $M$5 on CIFAR-10 and $M$6 on CIFAR-100 (Huang et al., 2017).

Representative CIFAR-scale results are substantial. For DenseNet-100, CIFAR-10 error is reduced from $M$7 for the single model to $M$8, and CIFAR-100 from $M$9 to $M$0. For Wide-ResNet-32, CIFAR-10 falls from $M$1 to $M$2, CIFAR-100 from $M$3 to $M$4, SVHN from $M$5 to $M$6, and Tiny ImageNet from $M$7 to $M$8 (Huang et al., 2017). The gains are reported as especially pronounced on harder datasets such as CIFAR-100 and Tiny ImageNet, where disagreement among accurate models is more useful (Huang et al., 2017).

The method also works at ImageNet scale, but with a more evident cycle-length trade-off. Using ResNet-50 and a standard 90-epoch budget, $M$9 cycles reduce top-1 validation error from $T$0 to $T$1, whereas $T$2 gives $T$3, suggesting that too many cycles leave too little time per cycle for adequate convergence (Huang et al., 2017). A similar pattern appears in the cycle-count study on DenseNet-40 for CIFAR-100: with total budget fixed at 300 epochs and $T$4, test error is $T$5 at $T$6, $T$7 at $T$8, $T$9 at $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$0, $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$1 at $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$2, and $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$3 at $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$4 (Huang et al., 2017).

This empirical profile establishes a central practical rule: snapshot ensembles are effective when each cycle is long enough to yield a strong member model. The approach is therefore not merely “save checkpoints and average”; it depends on a schedule that alternates between convergence and controlled escape (Huang et al., 2017).

3. Diversity, geometry, and basin structure

The original justification is that effective ensembling requires each member to be reasonably accurate while different members make different errors. Snapshot Ensembles attempt to satisfy both conditions by combining low-learning-rate convergence within each cycle with high-learning-rate restarts between cycles (Huang et al., 2017). The paper’s diversity analysis probes whether cyclic snapshots occupy meaningfully different minima by linearly interpolating between parameter vectors. For cyclic snapshots, many interpolations exhibit a non-convex error barrier, suggesting distinct local minima; for non-cyclic snapshots, later models often interpolate smoothly, indicating that they remain in essentially the same basin and offer less ensemble diversity (Huang et al., 2017).

The same study also reports lower pairwise correlations of softmax outputs for cyclic snapshots than for no-cycle snapshots, supporting the claim that cyclic restarts create diverse but still accurate predictors (Huang et al., 2017). This interpretation has since been refined rather than abandoned.

In transfer learning, the relevant geometry is the “pre-train basin”: models fine-tuned from the same checkpoint typically remain in the same basin of low loss. Work on transfer-learning ensembles shows that better exploration within this basin can improve ensembles trained from one checkpoint, but leaving the basin degrades both individual model quality and ensemble quality because the models lose the benefits of transfer learning (Sadrtdinov et al., 2023). In that setting, standard sequential Snapshot Ensembles can look strong for small ensembles, but later cycles drift farther from the first fine-tuned solution and may degrade. The proposed StarSSE modification instead launches additional exploratory runs from the first fine-tuned model, producing a “star” geometry rather than a chain, and reports stronger ensembles and stronger uniform model soups than standard SSE in those experiments (Sadrtdinov et al., 2023).

Interpretability-based diversity analysis points in the same general direction but at a different representational level. Using a shared ImageNet-pretrained Inception parent, one study compares Snapshot Ensembles with Prune and Tune Ensembles and finds that snapshot checkpoints appear more highly correlated with the parent and less internally differentiated than PAT children, both in feature visualizations and in saliency-map comparisons (Whitaker et al., 2023). This does not imply that snapshot members are redundant; it suggests that their diversity ceiling is constrained by shared ancestry, shared topology, and a single optimization trajectory (Whitaker et al., 2023).

4. Weighting, pruning, and theoretical reinterpretations

A major later development is the observation that snapshots collected along one training path are not equally useful. “Effective training-time stacking” argues that vanilla equal-weight averaging is suboptimal because later snapshots are often stronger, while neighboring snapshots may be highly similar. It replaces plain averaging with selection and weighting based on training-time loss or likelihood, without fitting a separate validation-based meta-learner (Proscura et al., 2022).

On CIFAR-10, this changes the behavior measurably. The paper reports $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$5 for “min, eq” versus $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$6 for “min, stack,” and $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$7 for “middle, stack,” compared with $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$8 for the single model and $$\alpha(t) = f\!\left(\mathrm{mod}\!\left(t-1,\left\lceil T/M \right\rceil\right)\right),$$9 for an independent ensemble of five models (Proscura et al., 2022). The same paper also reports that training-likelihood and validation-likelihood weighting are essentially tied in its setup, supporting its claim that a separate validation-driven stacking stage is unnecessary for snapshot-style ensembling (Proscura et al., 2022).

Another line of work adapts snapshot ideas to periodic retraining and cross-generation stability rather than single-run accuracy. “Wisdom of the Ensemble” defines consistency and correct-consistency, proves that an ensemble learner’s consistency and correct-consistency are not less than the average of individual learners under its geometric formulation, and proposes DynSnap: a dynamic snapshot ensemble that combines snapshot learning, repeated retraining diversity, and pruning by validation accuracy (Wang et al., 2020). DynSnap-cyc uses the cosine cyclic schedule and saves the best learner in each cycle according to validation accuracy, then prunes snapshots using a threshold of the form $t$0 (Wang et al., 2020). On CIFAR-100 with ResNet56 and AVG aggregation, the reported numbers are Snapshot ACC $t$1, CON $t$2, ACC-CON $t$3 versus DynSnap-cyc ACC $t$4, CON $t$5, ACC-CON $t$6 (Wang et al., 2020).

Robustness-focused work has also adopted a simplified checkpoint-ensemble perspective. “Improving Adversarial Robustness for Free with Snapshot Ensemble” does not seek multiple local minima with cosine restarts; instead, it stores the last few parameter checkpoints from the end of a training run and ensembles them. The paper describes this as simpler than the original method and reports a $t$7 to $t$8 increase in accuracy when compared to traditional adversarial training, depending on hyperparameters and datasets (Wang, 2021). A plausible implication is that the core snapshot intuition can survive even when the local-minima-harvesting mechanism is weakened, although that paper does not directly benchmark against the original Huang et al. procedure (Wang, 2021).

5. Generalizations beyond standard image classification

Snapshot ensembling has become a portable template rather than a task-specific trick. Later papers preserve the one-run, multi-checkpoint logic but redefine what counts as a snapshot, how snapshots are aggregated, or what role snapshots play in later training.

Variant	Main mechanism	Reported outcome
Training-time stacking	Selects and weights snapshots by training-time likelihood or loss	“middle, stack” reaches $t$9 on CIFAR-10 (Proscura et al., 2022)
StarSSE	Starts each additional run from the first fine-tuned model	Stronger ensembles and soups than standard SSE in transfer learning (Sadrtdinov et al., 2023)
SnapE	Uses snapshots for KGE link prediction and snapshot-guided hard negatives	Constantly outperforms the single model approach while keeping training time constant (Shaban et al., 2024)
Snapshot Spectral Clustering	Saves encoder snapshots and fuses landmark affinities for deep clustering	On CIFAR-10, the ensemble performs over twice as good as the base model (Piróg et al., 2023)
Symbolic Snapshot Ensembles	Saves intermediate ILP hypotheses from one run and combines them with MDL weighting	Predictive accuracy improves by $T$0 with less than $T$1 computational overhead (Liu et al., 28 Oct 2025)

In knowledge-graph link prediction, SnapE transfers the snapshot idea to KGE models such as TransE, DistMult, ComplEx, and RotatE. The method uses cyclic or deferred cyclic schedules, saves snapshots at learning-rate minima, and—crucially—lets earlier snapshots generate hard negative examples for later cycles. The abstract states that evaluation with four base models across four datasets constantly outperforms the single-model approach while keeping training time constant (Shaban et al., 2024).

In unsupervised learning, Snapshot Spectral Clustering replaces prediction averaging with representation-level fusion. A single convolutional autoencoder is trained with a cosine scheduler; encoder snapshots are saved after $T$2 epochs, each snapshot produces an embedding, each embedding yields a landmark-based affinity matrix, and the fused affinities are clustered by a spectral procedure (Piróg et al., 2023). The paper presents this as a costless or nearly costless way to generate deep clustering ensembles, and reports particularly strong gains on CIFAR-10 (Piróg et al., 2023).

The 2025 symbolic extension shows that the idea is not restricted to neural networks. “Symbolic Snapshot Ensembles” applies the one-run, many-intermediates template to inductive logic programming by saving intermediate hypotheses from a single ILP run and combining them using a minimum description length weighting scheme. The abstract reports a $T$3 improvement in predictive accuracy with less than $T$4 computational overhead on multiple benchmarks, including game playing and visual reasoning (Liu et al., 28 Oct 2025).

6. Limitations, misconceptions, and competing fast-ensemble designs

The slogan “Train 1, get $T$5 for free” is accurate only in a restricted sense. Relative to a standard single-model run, the training time is unchanged because the total number of optimization steps is the same, but inference with $T$6 snapshots costs about $T$7 forward passes and requires storing $T$8 checkpoints (Huang et al., 2017). Snapshot Ensembles therefore shift the ensemble burden from training to testing, rather than eliminating it.

Architectural scope is another limitation. Because snapshots come from one base network, they assume that ensemble members share the same architecture. MotherNets emphasizes this point explicitly, noting that Snapshot Ensembles “cannot be applied to state-of-the-art diverse ensembles” containing arbitrary architectures with structural differences (Wasay et al., 2018). That paper treats SE as the strongest prior fast baseline and as a valid point on the same Pareto frontier, but reports that MotherNets can reduce absolute test error by $T$9 to $M$0 percent and be up to $M$1 faster in training compared to Snapshot Ensembles, while also supporting heterogeneous architectures (Wasay et al., 2018).

The number of snapshots is not monotone in benefit. The original ImageNet result already shows degradation from $M$2 to $M$3 under a fixed 90-epoch budget (Huang et al., 2017). MotherNets argues more broadly that as ensemble size grows, Snapshot Ensembles can become worse by more than $M$4 in a 100-model VGG experiment on CIFAR-10, and links this to reduced diversity and weaker individual member quality (Wasay et al., 2018). Transfer-learning results reinforce the same caution: forcing exploration too far away from the pre-train basin increases diversity, but degrades both member quality and ensemble quality (Sadrtdinov et al., 2023).

The method’s uncertainty behavior is also context-dependent. In an industrial image-classification study on SIP-17 with LeNet-5, Snapshot Ensembles are cheap to train but underperform strongly on in-distribution predictive quality: $M$5 accuracy and $M$6 NLL, versus $M$7 and $M$8 for a single model, $M$9 and $\lceil T/M \rceil$0 for a deep ensemble, and $\lceil T/M \rceil$1 and $\lceil T/M \rceil$2 for a batch ensemble (Thuy et al., 2024). That paper further reports that Snapshot Ensembles have surprisingly high total uncertainty on the in-distribution set and weaker diversity quality than batch and deep ensembles, even though they can perform reasonably in classification-with-rejection curves (Thuy et al., 2024).

A final limitation is memory efficiency at deployment. PFGE frames Snapshot Ensembles and FGE as having solved training-time cost but not test-time storage cost, because many distinct checkpoints must still be retained and evaluated. Its remedy is a lightweight ensemble of higher-performing models produced by successive SWA procedures; under the paper’s setup, PFGE uses $\lceil T/M \rceil$3 models instead of $\lceil T/M \rceil$4, implying $\lceil T/M \rceil$5 of the memory overhead of FGE or SWAG and therefore $\lceil T/M \rceil$6 memory efficiency (Guo et al., 2022). This suggests that snapshot-style ensembling is best viewed not as a final answer to efficient ensembling, but as one influential point in a larger design space of one-run, low-cost, and partially compressed ensemble methods (Guo et al., 2022).