--- title: 'Sequential Ensembling: Methods & Results' url: https://www.emergentmind.com/topics/sequential-ensembling type: topic --- # Sequential Ensembling: Methods & Results Searching arXiv for recent and foundational papers on sequential ensembling across domains. Sequential ensembling denotes a family of ensemble constructions in which the contribution of multiple models is ordered rather than purely parallel. Across the literature, that order may arise during training, as later learners are conditioned on residuals, uncertainties, or checkpoints from earlier learners; during inference, as models are invoked one after another with adaptive halting or routing; or over structured inputs, where classifier outputs are fused across timesteps, frames, tokens, or experts. The topic therefore spans classical deep ensembling, boosting-style cascades, sequential Monte Carlo over language-model ensembles, expert chains for continual learning, and single-run snapshot or tangent-space compositions that seek ensemble gains without full parallel cost [2005.00570]. ## 1. Conceptual scope and problem setting A unifying feature of sequential ensembling is that the ensemble state after stage \(t\) affects what happens at stage \(t+1\). In some systems, this state is an accumulated prediction, as in frame-wise temporal fusion or token-level probability aggregation. In others, it is a learned routing or halting signal, a residual representation, or a parameter-space perturbation. This distinguishes sequential ensembling from naive bagging or static averaging, where all members are trained and queried independently and combined only at the end. The literature uses this sequentiality in several non-equivalent senses. Inference can be sequential over models, as in adaptive per-sample halting [2301.12378]. It can be sequential over observations, as in time-distributed deep ensembles for tracked image sequences [2309.11333]. It can be sequential over latent decisions in text generation, where the next token or byte is produced under an evolving ensemble state [2406.12585]. It can also be sequential in training, where later learners explicitly target what earlier learners left unexplained, as in residual boosting for sparse autoencoders [2505.16077]. A practical motivation is compute allocation. Classical evidence already showed that ensembling smaller models can outperform a single larger model with both higher accuracy and fewer total FLOPs, while also exposing a flexible speed–accuracy trade-off and easier parallelization [2005.00570]. Sequential variants generalize that idea by deciding *when* additional ensemble members are needed, *where* in a sequence they should be used, or *how* later models should specialize. | Mode of sequentiality | Representative mechanism | Representative papers | |---|---|---| | Training-stage sequencing | residual fitting, uncertainty weighting, snapshot stacking | [2505.16077], [2104.10715], [2206.13491] | | Inference-stage sequencing | adaptive halting, temporal spreading, token routing | [2301.12378], [2309.11333], [2605.00419] | | Sequence-structured aggregation | HMM decoding, byte-level SMC | [1906.09356], [2603.05432] | | Model-space composition | sparse subnetwork cycles, tangent-vector composition, expert chains | [2206.00794], [2307.08114], [2504.06664] | ## 2. Training-stage sequencing: boosting, stacking, and residual specialization One major lineage treats sequential ensembling as a training protocol in which later components are conditioned on signals from earlier ones. In uncertainty-aware multi-modal boosting, learner \(h^{j+1}\) is trained on modality \(j+1\) with per-sample loss weighted by the uncertainty of learner \(h^j\), rather than by raw loss. The per-sample objective is \[ L_{n}^{j+1} = \text{loss}( h^{j+1}(\mathbf{x}_n^{j+1}), y_n ) \cdot \sigma_{h^j}(\mathbf{x}_n^j), \] and the final prediction can be combined by uncertainty-weighted averaging, so lower-uncertainty predictions contribute more strongly [2104.10715]. This replaces classical loss-focused boosting with uncertainty propagation across modalities and was reported to decrease overall entropy, improve calibration, and yield better prediction intervals in healthcare tasks. Residual specialization appears in boosted sparse autoencoder ensembles. There, the \(j\)-th SAE is trained not on the original activation \(x^{(n)}\) but on the residual \[ r^{(n, j)} = \begin{cases} x^{(n)} & \text{if } j=1 \\ x^{(n)} - \sum_{\ell=1}^{j-1} g(r^{(n, \ell)}; \theta^{(\ell)}) & \text{if } j>1, \end{cases} \] with objective \[ \mathcal{L}_{\text{Boost}} = \frac{1}{N} \sum_{n=1}^{N} \left[ \| r^{(n, j)} - g(r^{(n, j)}; \theta^{(j)}) \|_{2}^{2} + \lambda \| z^{(n, j)} \|_{p} \right]. \] The ensemble reconstruction sums sequentially trained residual reconstructions. Reported effects include better reconstruction of language-model activations, more diverse features, improved stability, and stronger downstream concept detection and spurious-correlation removal than single SAEs [2505.16077]. A different training-stage form of sequential ensembling uses a single optimization trajectory as the source of multiple ensemble members. Effective training-time stacking improves snapshot ensembling by selecting and weighting snapshots collected along one run. Instead of uniform averaging, predictions take the form \[ \hat{f}(x) = \frac{1}{N} \sum_{k=1}^N w_k f_{\theta_k}(x), \] where weights are derived from loss or likelihood and may be temperature-scaled. On CIFAR-10, the reported single-model accuracy was \(89.49\%\), unweighted snapshot ensembling with 24 models reached \(89.9\%\), weighted/stacked snapshot ensembling reached up to \(90.5\%\), and the best stacking approach with mid-cycle models reached \(90.8\%\) [2206.13491]. The same single-run intuition underlies SuperNet, which treats checkpoints from one DNN trajectory as ensemble members and trains only an aggregation layer over their outputs. If \(f^{(i)}\) are the saved sub-models, SuperNet predicts \[ \mathbf{p}_\text{SuperNet}(x) = \sigma\left( \sum_{i=1}^K \alpha_i \mathbf{p}^{(i)} + \mathbf{b} \right), \] with cross-entropy on the aggregation layer while base sub-models remain frozen [2003.13021]. This suggests a broad principle: training-stage sequential ensembling often trades maximal diversity for sharply reduced training cost, then reintroduces selectivity through weighting, residual fitting, or a learned combiner. ## 3. Inference-stage sequencing: adaptive halting and temporal spreading Another major lineage keeps base models fixed and makes the ensemble itself sequential at test time. Inference-efficient ensemble learning treats the ensemble as a sequential process. After each model, a common selector predicts a conditional halting probability \(h_t\); the probability that halting occurs exactly at step \(t\) is \[ p_t = h_t \prod_{i=1}^{t-1} (1 - h_i). \] The selector and base models are jointly optimized using ensemble-performance, cost, and ranking terms, so easy samples stop early while hard samples continue deeper into the ensemble [2301.12378]. Reported results include up to \(56\%\) inference cost reduction while maintaining comparable performance to the full ensemble, and on CIFAR-10 with ResNet-18 an average of \(1.32\) models used per sample instead of \(3.0\), with top-1 accuracy \(95.81\%\) versus \(95.62\%\) for the full ensemble. Sequentiality can also be distributed across the *input* sequence instead of across a model list per sample. Deep Ensembles Spread Over Time applies only one ensemble member to each frame and fuses predictions over time. For sequence \(\mathbf{x}=\{x^1,\dots,x^T\}\) with assignment \(\sigma(t)\), DESOT predicts \[ \mathbf{p}_{\text{DESOT}}(\mathbf{x}) = \frac{1}{T} \sum_{t=1}^{T} \mathbf{p}_{\sigma(t)}(x^t). \] This uses exactly \(T\) forward passes for a sequence, rather than \(M \cdot T\) for a standard \(M\)-member deep ensemble [2309.11333]. On traffic sign classification with \(T=11\), ResNet-18, and \(M=5\), the reported metrics were: single model \(97.34\) accuracy, \(81.12\) F1, \(0.0124\) Brier, \(2.88\) ECE; DESOT\(_5\) \(97.60\), \(83.26\), \(0.0108\), \(2.25\); and DE\(_5\) \(97.64\), \(82.73\), \(0.0108\), \(2.25\) [2309.11333]. The abstract of a semantic-segmentation paper extends this idea into a cascade of models conditioned on previous-stage class probabilities. It describes a method inspired by boosting that trains a cascade conditioned on class probabilities predicted by the previous model as an additional input, introduces an ADaptive modulatiON block for spatial feature modulation, and states that dynamic computation offloading can aid deployment on mobile devices [2210.05387]. A plausible implication is that sequential ensembling can be interpreted not only as output aggregation but also as staged feature conditioning. A recurring theme in this branch is conditional compute. Sequential inference is valuable precisely because many samples or timesteps do not require the full ensemble. The gain is not simply fewer arithmetic operations; it is a reallocation of ensemble capacity to samples, frames, or regions where additional diversity matters most. ## 4. Sequential dependence in the data: HMM ensembles and recommendation Sequential ensembling also refers to settings where the *data* are dependent and the combiner must model that dependence explicitly. In unsupervised ensemble classification with sequential data, the unknown label sequence \(\mathbf{y}=[y_1,\dots,y_N]\) is modeled as a first-order time-homogeneous Markov chain with transition matrix \(\mathbf{T}\), while the responses of \(M\) classifiers are conditionally independent given the true label [1906.09356]. The joint label probability is \[ \Pr(\mathbf{y}=\mathbf{k}) = \Pr(y_1=k_1)\prod_{n=2}^{N} T(k_n,k_{n-1}), \] and the overall model is an ensemble HMM. Parameter estimation proceeds through moment matching and Expectation Maximization. Moment matching exploits relations such as \[ \mathbb{E}[\mathbf{f}_m(X)] = \mathbf{\Gamma}_m \bm{\pi} \] and time-lagged cross-correlations, then EM refines parameters using Baum–Welch updates. Decoding uses Viterbi sequence MAP estimation. The reported conclusion is that methods explicitly modeling sequence dependence notably outperform i.i.d.-based stacking, majority vote, and single-best selection, with gains on synthetic data and on POS tagging, NER, and biomedical information extraction [1906.09356]. This is a distinct meaning of sequential ensembling: the ensemble is not merely invoked in sequence; it is itself a probabilistic model over sequential labels. Sequential recommendation provides a different example. EnsRec trains an ID-based SR model and a text-based SR model independently, then combines them by summing item scores at inference: \[ s_{\text{EnsRec},u,k}(i) = s_{\text{ID},u,k}(i) + s_{\text{text},u,k}(i). \] The authors argue that ID and text models learn complementary signals, quantified through low Jaccard overlap of well-modeled users and a high Genie upper bound, and report that simple ensembling outperforms several complex fusion baselines on Beauty, Sports, Toys, and Steam [2512.17820]. Although the ensemble rule itself is simple, the setting is intrinsically sequential because the base predictors are sequential recommendation models. These cases show that sequential ensembling need not imply a chain of experts or a progressive deep cascade. It may instead designate an ensemble for *sequentially dependent targets*, where explicit temporal structure in the combiner is the central ingredient. ## 5. Token-level and string-level sequential ensembling in language models Large language models have made token-by-token sequential ensembling a central contemporary variant. One direct approach treats token generation as classification. Generation as Classification first forms the union vocabulary \[ V^U = \bigcup_{i=1}^n V^i, \] maps each model’s next-token distribution into that shared space with a mapping matrix, and averages: \[ q(\cdot) = \frac{1}{n} \sum_{i=1}^{n} p^i(\cdot\,|\,\mathcal{I}^{\,i}) \cdot \mathbf{M}^{i}. \] The selected token is then re-tokenized and appended for each model [2406.12585]. The same work also proposes a thresholded cascade, where a gate model’s confidence determines whether full ensembling is invoked, and reports that ensembling only \(7\%\)–\(10\%\) of tokens can reduce latency substantially without sacrificing accuracy. Selective ensembling becomes more subtle in long-form generation because ensembling at every token can degrade performance. SAFE identifies two factors that determine where ensembling is appropriate: tokenization mismatch across models and consensus in next-token distributions [2510.15346]. It uses a Generate–Verify–Ensemble cycle. Ensembling is forbidden immediately after an OOV-like token for any verifier, and also skipped when all models already agree or when the average probability of the aligned token exceeds \(0.5\). SAFE further introduces probability sharpening by consolidating probability mass from subword tokens sharing a prefix into a representative token. The reported experiments show gains on MATH500 and BBH even when ensembling fewer than \(1\%\) of tokens. A more radical efficiency result is the Mixture-model-like Ensemble. Instead of explicitly averaging all next-token distributions, it samples a single model index \(i\) at each step according to weights \(\lambda_i\), performs one forward pass, and samples from that model. Because \[ P_{ME}(y \mid x_{