Sequential Ensembling: Methods & Results

• Sequential ensembling is a family of methods that orders model predictions so that later stages correct previous outputs using residuals, uncertainties, or adaptive routing.
• Training-stage sequencing techniques like uncertainty-aware boosting and snapshot stacking enhance accuracy and calibration while reducing computational costs.
• Inference-stage strategies—such as adaptive halting and token-level aggregation—optimize ensemble usage to achieve robust performance in diverse applications.

Searching arXiv for recent and foundational papers on sequential ensembling across domains. Sequential ensembling denotes a family of ensemble constructions in which the contribution of multiple models is ordered rather than purely parallel. Across the literature, that order may arise during training, as later learners are conditioned on residuals, uncertainties, or checkpoints from earlier learners; during inference, as models are invoked one after another with adaptive halting or routing; or over structured inputs, where classifier outputs are fused across timesteps, frames, tokens, or experts. The topic therefore spans classical deep ensembling, boosting-style cascades, sequential Monte Carlo over language-model ensembles, expert chains for continual learning, and single-run snapshot or tangent-space compositions that seek ensemble gains without full parallel cost (Kondratyuk et al., 2020).

1. Conceptual scope and problem setting

A unifying feature of sequential ensembling is that the ensemble state after stage $t$ affects what happens at stage $t+1$. In some systems, this state is an accumulated prediction, as in frame-wise temporal fusion or token-level probability aggregation. In others, it is a learned routing or halting signal, a residual representation, or a parameter-space perturbation. This distinguishes sequential ensembling from naive bagging or static averaging, where all members are trained and queried independently and combined only at the end.

The literature uses this sequentiality in several non-equivalent senses. Inference can be sequential over models, as in adaptive per-sample halting (Li et al., 2023). It can be sequential over observations, as in time-distributed deep ensembles for tracked image sequences (Meding et al., 2023). It can be sequential over latent decisions in text generation, where the next token or byte is produced under an evolving ensemble state (Yu et al., 2024). It can also be sequential in training, where later learners explicitly target what earlier learners left unexplained, as in residual boosting for sparse autoencoders (Gadgil et al., 21 May 2025).

A practical motivation is compute allocation. Classical evidence already showed that ensembling smaller models can outperform a single larger model with both higher accuracy and fewer total FLOPs, while also exposing a flexible speed–accuracy trade-off and easier parallelization (Kondratyuk et al., 2020). Sequential variants generalize that idea by deciding when additional ensemble members are needed, where in a sequence they should be used, or how later models should specialize.

Mode of sequentiality	Representative mechanism	Representative papers
Training-stage sequencing	residual fitting, uncertainty weighting, snapshot stacking	(Gadgil et al., 21 May 2025, Sarawgi et al., 2021, Proscura et al., 2022)
Inference-stage sequencing	adaptive halting, temporal spreading, token routing	(Li et al., 2023, Meding et al., 2023, Fu et al., 1 May 2026)
Sequence-structured aggregation	HMM decoding, byte-level SMC	(Traganitis et al., 2019, Chan et al., 5 Mar 2026)
Model-space composition	sparse subnetwork cycles, tangent-vector composition, expert chains	(Jantre et al., 2022, Liu et al., 2023, Wang et al., 9 Apr 2025)

2. Training-stage sequencing: boosting, stacking, and residual specialization

One major lineage treats sequential ensembling as a training protocol in which later components are conditioned on signals from earlier ones. In uncertainty-aware multi-modal boosting, learner $h^{j+1}$ is trained on modality $j+1$ with per-sample loss weighted by the uncertainty of learner $h^j$, rather than by raw loss. The per-sample objective is

$$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$

and the final prediction can be combined by uncertainty-weighted averaging, so lower-uncertainty predictions contribute more strongly (Sarawgi et al., 2021). This replaces classical loss-focused boosting with uncertainty propagation across modalities and was reported to decrease overall entropy, improve calibration, and yield better prediction intervals in healthcare tasks.

Residual specialization appears in boosted sparse autoencoder ensembles. There, the $j$-th SAE is trained not on the original activation $x^{(n)}$ but on the residual

$$r^{(n, j)} = \begin{cases} x^{(n)} & \text{if } j=1 \ x^{(n)} - \sum_{\ell=1}^{j-1} g(r^{(n, \ell)}; \theta^{(\ell)}) & \text{if } j>1, \end{cases}$$

with objective

$$\mathcal{L}_{\text{Boost}} = \frac{1}{N} \sum_{n=1}^{N} \left[ \| r^{(n, j)} - g(r^{(n, j)}; \theta^{(j)}) \|_{2}^{2} + \lambda \| z^{(n, j)} \|_{p} \right].$$

The ensemble reconstruction sums sequentially trained residual reconstructions. Reported effects include better reconstruction of language-model activations, more diverse features, improved stability, and stronger downstream concept detection and spurious-correlation removal than single SAEs (Gadgil et al., 21 May 2025).

A different training-stage form of sequential ensembling uses a single optimization trajectory as the source of multiple ensemble members. Effective training-time stacking improves snapshot ensembling by selecting and weighting snapshots collected along one run. Instead of uniform averaging, predictions take the form

$t+1$0

where weights are derived from loss or likelihood and may be temperature-scaled. On CIFAR-10, the reported single-model accuracy was $t+1$1, unweighted snapshot ensembling with 24 models reached $t+1$2, weighted/stacked snapshot ensembling reached up to $t+1$3, and the best stacking approach with mid-cycle models reached $t+1$4 (Proscura et al., 2022).

The same single-run intuition underlies SuperNet, which treats checkpoints from one DNN trajectory as ensemble members and trains only an aggregation layer over their outputs. If $t+1$5 are the saved sub-models, SuperNet predicts

$t+1$6

with cross-entropy on the aggregation layer while base sub-models remain frozen (Bukowski et al., 2020). This suggests a broad principle: training-stage sequential ensembling often trades maximal diversity for sharply reduced training cost, then reintroduces selectivity through weighting, residual fitting, or a learned combiner.

3. Inference-stage sequencing: adaptive halting and temporal spreading

Another major lineage keeps base models fixed and makes the ensemble itself sequential at test time. Inference-efficient ensemble learning treats the ensemble as a sequential process. After each model, a common selector predicts a conditional halting probability $t+1$7; the probability that halting occurs exactly at step $t+1$8 is

$t+1$9

The selector and base models are jointly optimized using ensemble-performance, cost, and ranking terms, so easy samples stop early while hard samples continue deeper into the ensemble (Li et al., 2023). Reported results include up to $h^{j+1}$0 inference cost reduction while maintaining comparable performance to the full ensemble, and on CIFAR-10 with ResNet-18 an average of $h^{j+1}$1 models used per sample instead of $h^{j+1}$2, with top-1 accuracy $h^{j+1}$3 versus $h^{j+1}$4 for the full ensemble.

Sequentiality can also be distributed across the input sequence instead of across a model list per sample. Deep Ensembles Spread Over Time applies only one ensemble member to each frame and fuses predictions over time. For sequence $h^{j+1}$5 with assignment $h^{j+1}$6, DESOT predicts

$h^{j+1}$7

This uses exactly $h^{j+1}$8 forward passes for a sequence, rather than $h^{j+1}$9 for a standard $j+1$0-member deep ensemble (Meding et al., 2023). On traffic sign classification with $j+1$1, ResNet-18, and $j+1$2, the reported metrics were: single model $j+1$3 accuracy, $j+1$4 F1, $j+1$5 Brier, $j+1$6 ECE; DESOT$j+1$7 $j+1$8, $j+1$9, $h^j$0, $h^j$1; and DE$h^j$2 $h^j$3, $h^j$4, $h^j$5, $h^j$6 (Meding et al., 2023).

The abstract of a semantic-segmentation paper extends this idea into a cascade of models conditioned on previous-stage class probabilities. It describes a method inspired by boosting that trains a cascade conditioned on class probabilities predicted by the previous model as an additional input, introduces an ADaptive modulatiON block for spatial feature modulation, and states that dynamic computation offloading can aid deployment on mobile devices (Khirodkar et al., 2022). A plausible implication is that sequential ensembling can be interpreted not only as output aggregation but also as staged feature conditioning.

A recurring theme in this branch is conditional compute. Sequential inference is valuable precisely because many samples or timesteps do not require the full ensemble. The gain is not simply fewer arithmetic operations; it is a reallocation of ensemble capacity to samples, frames, or regions where additional diversity matters most.

4. Sequential dependence in the data: HMM ensembles and recommendation

Sequential ensembling also refers to settings where the data are dependent and the combiner must model that dependence explicitly. In unsupervised ensemble classification with sequential data, the unknown label sequence $h^j$7 is modeled as a first-order time-homogeneous Markov chain with transition matrix $h^j$8, while the responses of $h^j$9 classifiers are conditionally independent given the true label (Traganitis et al., 2019). The joint label probability is

$$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$0

and the overall model is an ensemble HMM.

Parameter estimation proceeds through moment matching and Expectation Maximization. Moment matching exploits relations such as

$$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$1

and time-lagged cross-correlations, then EM refines parameters using Baum–Welch updates. Decoding uses Viterbi sequence MAP estimation. The reported conclusion is that methods explicitly modeling sequence dependence notably outperform i.i.d.-based stacking, majority vote, and single-best selection, with gains on synthetic data and on POS tagging, NER, and biomedical information extraction (Traganitis et al., 2019). This is a distinct meaning of sequential ensembling: the ensemble is not merely invoked in sequence; it is itself a probabilistic model over sequential labels.

Sequential recommendation provides a different example. EnsRec trains an ID-based SR model and a text-based SR model independently, then combines them by summing item scores at inference: $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$2 The authors argue that ID and text models learn complementary signals, quantified through low Jaccard overlap of well-modeled users and a high Genie upper bound, and report that simple ensembling outperforms several complex fusion baselines on Beauty, Sports, Toys, and Steam (Collins et al., 19 Dec 2025). Although the ensemble rule itself is simple, the setting is intrinsically sequential because the base predictors are sequential recommendation models.

These cases show that sequential ensembling need not imply a chain of experts or a progressive deep cascade. It may instead designate an ensemble for sequentially dependent targets, where explicit temporal structure in the combiner is the central ingredient.

5. Token-level and string-level sequential ensembling in LLMs

LLMs have made token-by-token sequential ensembling a central contemporary variant. One direct approach treats token generation as classification. Generation as Classification first forms the union vocabulary

$$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$3

maps each model’s next-token distribution into that shared space with a mapping matrix, and averages: $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$4 The selected token is then re-tokenized and appended for each model (Yu et al., 2024). The same work also proposes a thresholded cascade, where a gate model’s confidence determines whether full ensembling is invoked, and reports that ensembling only $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$5–$$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$6 of tokens can reduce latency substantially without sacrificing accuracy.

Selective ensembling becomes more subtle in long-form generation because ensembling at every token can degrade performance. SAFE identifies two factors that determine where ensembling is appropriate: tokenization mismatch across models and consensus in next-token distributions (Yun et al., 17 Oct 2025). It uses a Generate–Verify–Ensemble cycle. Ensembling is forbidden immediately after an OOV-like token for any verifier, and also skipped when all models already agree or when the average probability of the aligned token exceeds $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$7. SAFE further introduces probability sharpening by consolidating probability mass from subword tokens sharing a prefix into a representative token. The reported experiments show gains on MATH500 and BBH even when ensembling fewer than $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$8 of tokens.

A more radical efficiency result is the Mixture-model-like Ensemble. Instead of explicitly averaging all next-token distributions, it samples a single model index $$L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j),$$9 at each step according to weights $j$0, performs one forward pass, and samples from that model. Because

$j$1

this is mathematically equivalent to sampling from the conventional ensemble distribution, but only for stochastic sampling and not for greedy decoding (Fu et al., 1 May 2026). Reported speedups are $j$2–$j$3 over conventional ensemble decoding.

The most principled global treatment replaces local token averaging with string-level ensemble distributions. For $j$4 LLMs and aggregation function $j$5, the unnormalized $j$6-ensemble is

$j$7

with normalized distribution $j$8. Because exact sampling is intractable, byte-level Sequential Monte Carlo is used to sample in a shared character space, which also handles mismatching tokenizations (Chan et al., 5 Mar 2026). The paper studies generalized means, including product, sum, min, and max ensembles, and reports that consensus-seeking variants such as product and min often outperform averaging on structured generation tasks. This yields an important correction to a common assumption: locally normalized next-token averaging is not the same as sampling from the intended global ensemble over strings.

6. Model-space composition, sparse subnetworks, and continual expert chains

A separate branch moves the ensemble from output space into model space. Sequential Bayesian Neural Subnetwork Ensembles use a spike-and-slab Bayesian prior to maintain sparse subnetworks and a two-phase schedule: one exploration phase with a large constant learning rate followed by multiple exploitation phases. After each exploitation phase, variational means are perturbed by

$j$9

then quickly reconverge to a new minimum (Jantre et al., 2022). The reported CIFAR-10 results include accuracy $x^{(n)}$0 and NLL $x^{(n)}$1 for SeBayS-Freeze/NoFreeze, compared with $x^{(n)}$2 and $x^{(n)}$3 for a single sparse Bayesian network, while using one pass; diversity metrics also increase markedly for the no-freeze variant.

Tangent Model Composition linearizes each independently fine-tuned component around a common pre-trained point. The tangent space at weights $x^{(n)}$4 is

$x^{(n)}$5

and linear combinations of tangent models remain tangent models: $x^{(n)}$6 As a result, multiple component models can be composed before inference and executed at the cost of a single model (Liu et al., 2023). The paper reports an average $x^{(n)}$7 accuracy improvement over ensembling non-linearly fine-tuned models, with $x^{(n)}$8 to $x^{(n)}$9 lower inference cost, and emphasizes that components can be added, scaled, or subtracted for unlearning.

Continual fine-tuning motivates an expert-chain formulation. Sequential Ensemble of Experts removes a centralized router and lets each expert independently decide whether it should answer a query. A query is passed from newest expert to oldest; an expert emits a positive or negative indicator, and if all decline, the base model handles the query (Wang et al., 9 Apr 2025). New tasks require training only a new expert, not retraining the full system. Reported outcomes include near-zero backward transfer for SEE(10%), routing F1 above $$r^{(n, j)} = \begin{cases} x^{(n)} & \text{if } j=1 \ x^{(n)} - \sum_{\ell=1}^{j-1} g(r^{(n, \ell)}; \theta^{(\ell)}) & \text{if } j>1, \end{cases}$$0 with moderate rehearsal, and more than $$r^{(n, j)} = \begin{cases} x^{(n)} & \text{if } j=1 \ x^{(n)} - \sum_{\ell=1}^{j-1} g(r^{(n, \ell)}; \theta^{(\ell)}) & \text{if } j>1, \end{cases}$$1 of out-of-distribution instances routed to the base model.

These model-space and expert-chain methods show that sequential ensembling can be more than staged prediction combination. It can be a compositional principle over subnetworks, tangent deltas, or autonomous experts, with the ensemble state realized directly in parameters or routing behavior.

7. Recurring advantages, limitations, and common misconceptions

Several advantages recur across domains. Sequential ensembling often improves predictive performance, uncertainty estimation, calibration, or robustness while using less inference than full parallel ensembles. DESOT achieves ensemble-like uncertainty benefits at the compute cost of a single model per frame (Meding et al., 2023). IRENE reduces average model usage while keeping accuracy comparable to the full ensemble (Li et al., 2023). SeBayS and TMC report single-pass or single-model inference with ensemble gains in accuracy and uncertainty (Jantre et al., 2022, Liu et al., 2023). In LLMs, selective or stochastic token-level methods reduce the need for explicit full-distribution aggregation (Yun et al., 17 Oct 2025, Fu et al., 1 May 2026).

The literature also corrects several misconceptions. Sequential ensembling is not synonymous with classical boosting: uncertainty-aware modality boosting, residual SAE boosting, adaptive halting, token-level averaging, tangent composition, and SMC over string distributions are materially different constructions (Sarawgi et al., 2021, Gadgil et al., 21 May 2025, Li et al., 2023, Yu et al., 2024, Liu et al., 2023, Chan et al., 5 Mar 2026). Nor does ensemble quality necessarily require architectural diversity: one study found no statistically significant improvement from neural-architecture-search diversity, and reported that duplicating the most accurate architecture yielded the best results for a given resource budget (Kondratyuk et al., 2020).

Limitations are equally consistent. Some methods save compute but not memory: DESOT and standard deep ensembles still require storing all models (Meding et al., 2023). Some equivalences are decoding-dependent: the mixture-model-like LLM ensemble is equivalent to conventional ensembling for sampling, but not for greedy decoding (Fu et al., 1 May 2026). Some methods are sensitive to where ensembling occurs: SAFE argues that ensembling at every token can degrade long-form generation, especially under tokenization mismatch (Yun et al., 17 Oct 2025). Global language-model ensembling can also be substantially more expensive than local heuristics, since SMC operates at byte level and maintains particles (Chan et al., 5 Mar 2026). In continual and compositional settings, parameter growth or locality assumptions remain relevant: SEE adds experts over tasks (Wang et al., 9 Apr 2025), while TMC is designed for models local to a common pre-trained embedding (Liu et al., 2023).

Taken together, the research suggests that sequential ensembling is best understood not as a single algorithmic template but as an ordered resource-allocation principle. The ensemble may be distributed across training stages, samples, frames, tokens, or experts; it may live in output space, state space, or parameter space; and its central question is always the same: how should model diversity be exposed progressively so that later computation is spent only where earlier computation is insufficient?