Sequential Mixture of Experts (SeqMoE)

• Sequential Mixture of Experts (SeqMoE) is a research lineage of models that use ordered, iterative expert routing to gradually refine intermediate representations.
• Key methodologies include iterative gating, depth-wise routing, and hierarchical multi-level expert communication, which enhance model efficiency and performance.
• SeqMoE applications span diverse areas such as chemical reaction prediction, sequential recommendation, and online Bayesian inference, highlighting its practical relevance.

Sequential Mixture of Experts (SeqMoE) designates a family of mixture-of-experts formulations in which expert specialization is coupled to a sequential process rather than being exhausted by a single parallel routing event. Across the literature, the term is used for non-identical mechanisms: divide-and-conquer expert specialization for chemical reaction prediction, sparse expert routing inside sequential recommenders, iterative expert communication within a layer, depth-wise gating across successive blocks, and sequential Bayesian inference in time-evolving mixture models (Guo et al., 2023); (Pan et al., 29 Oct 2025); (Wang et al., 23 Jun 2025); (Cornebise et al., 2011). This suggests that SeqMoE is best understood as a research lineage of related designs rather than a single canonical architecture.

1. Conceptual scope and nomenclature

In conventional sparse MoE layers, experts operate independently in parallel, with a token-level output of the form $y=\sum_{i=1}^N g_i \cdot E_i(x)$, where only a small number of gates are active. "Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models" (Wang et al., 23 Jun 2025) defines its departure from this pattern by replacing one-shot routing with a $T$-step sequential relay inside a layer, so that the hidden state is updated iteratively as $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$, with a dedicated router at each iteration. In that formulation, sequentiality means that later experts consume the intermediate representation produced by earlier experts rather than a fixed initial input.

Other papers use the same label for different forms of ordered expert interaction. "Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography" (Bayatmakou et al., 23 Jul 2025) organizes experts in depth and applies a gate after each expert to mix the current expert output with the running representation from the previous layer. "Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts" (Pan et al., 29 Oct 2025), "Facet-Aware Multi-Head Mixture-of-Experts Model for Sequential Recommendation" (Liu et al., 2024), and "HyMoERec: Hybrid Mixture-of-Experts for Sequential Recommendation" (Li et al., 9 Nov 2025) embed MoE mechanisms inside sequential recommendation backbones, where the underlying task is sequential even when the expert computation at a given block is sparse rather than iterative. By contrast, "Adaptive sequential Monte Carlo by means of mixture of experts" (Cornebise et al., 2011), "Dynamic Mixture of Experts Models for Online Prediction" (Munezero et al., 2021), and "Horseshoe Mixtures-of-Experts (HS-MoE)" (Polson et al., 14 Jan 2026) use sequentiality in the statistical sense: inference proceeds online in time through particle filtering, particle learning, or online EM.

The resulting terminological landscape is heterogeneous. A SeqMoE system may therefore denote sequential expert communication, sequential routing in depth, MoE-enhanced sequential modeling, or sequential posterior updating. This is a central point for interpreting the literature.

2. Routing, gating, and expert composition

The most common SeqMoE mechanism remains input-dependent gating. In Fuxi-MME, the gating logits for an input vector $h\in\mathbb{R}^d$ are $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$, only the top-$k$ entries are retained, and the MoE output is $y=\sum_{i=1}^N g_i \cdot f_i(h)$ (Pan et al., 29 Oct 2025). The added noise term is used to encourage expert load balancing, while sparse top-$k$ activation keeps the compute cost proportional to $k\cdot d$ rather than $N\cdot d$.

Other SeqMoE models use hierarchical or multi-level routing. In FAME, each attention head is treated as a distinct facet, and each head contains its own $T$0-expert MoE for query generation. Expert outputs are aggregated through a router $T$1, giving $T$2; a second gate $T$3 then weights the heads themselves (Liu et al., 2024). The architecture is therefore a two-level mixture: first among experts within a facet, then among facets.

Hybrid formulations mix dense and sparse branches rather than only sparse experts. HyMoERec defines a shared dense expert $T$4, a specialized sparse expert set $T$5, and an Adaptive Expert Fusion rule $T$6, with $T$7 and a linear warm-up schedule $T$8 where $T$9 (Li et al., 9 Nov 2025). Mammo-Mamba uses an even smaller gating simplex: after each expert block, a two-layer gating MLP produces $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$0 with $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$1, and the new representation is $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$2 (Bayatmakou et al., 23 Jul 2025).

Statistical SeqMoE models use probabilistic experts rather than neural FFNs. In adaptive SMC, the proposal kernel is written as

$$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$3

where the $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$4 are logistic gating weights and each expert $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$5 belongs to a family of integrated curved exponential distributions (Cornebise et al., 2011). Here the gate partitions the ancestor space, and the experts model local proposal structure rather than feature transformations.

3. Reaction prediction and rare plausible outcomes

"Beyond the Typical: Modeling Rare Plausible Patterns in Chemical Reactions by Leveraging Sequential Mixture-of-Experts" (Guo et al., 2023) applies the SeqMoE idea to reaction prediction, defined there as predicting the outcome of a reaction from given reactants. The paper’s abstract states that generative models such as Transformer and VAE have typically been used for this task, but that these likelihood-maximization models overlook the inherent stochastic nature of chemical reactions, including the multiple ways electrons can be redistributed among atoms during the reaction process. When similar reactants can follow different electron redistribution patterns, the most common outcomes are typically predicted while less frequent but potentially crucial reaction patterns are neglected.

The proposed remedy is to organize the mapping space between reactants and electron redistribution patterns in a divide-and-conquer manner. Multiple expert models are trained, each specializing in a type of electron redistribution pattern in reaction. During inference, a dropout strategy is applied to each expert to improve electron redistribution diversity, and a ranking stage integrates predictions from multiple experts to produce the most plausible products. The reported experiments are conducted on the USPTO-MIT benchmark, described as the largest reaction prediction benchmark in the abstract, and the method is stated to outperform baselines (Guo et al., 2023).

The available description is precise about the divide-and-conquer motivation, expert specialization, expert dropout, and final ranking stage, but it does not provide gating equations, individual expert definitions, or sequential inference formulas for the SeqMoE mechanism itself (Guo et al., 2023). This makes the reaction-prediction paper notable both for its application focus and for the relative lack of low-level architectural detail in the available summary.

4. SeqMoE in sequential recommendation

Sequential recommendation has been one of the most active application areas for SeqMoE-style designs. In Fuxi-MME, the conventional single embedding matrix is decomposed into several lower-dimensional embedding matrices $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$6, each in $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$7. The Fuxi Block replaces its feed-forward network and gating projection with a sparse MoE layer containing $$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$8 experts, and only the top-$$x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})$$9 experts are activated for a token (Pan et al., 29 Oct 2025). The paper reports results on Amazon-Books, Amazon-Beauty, and Yelp using HR@10/50 and NDCG@10/50, and states that Fuxi-MME improves NDCG@10 on Amazon-Books by $h\in\mathbb{R}^d$0 over Fuxi-$h\in\mathbb{R}^d$1 and by $h\in\mathbb{R}^d$2–$h\in\mathbb{R}^d$3 over other baselines. Its ablations report that removing multi-embedding drops NDCG@10 by $h\in\mathbb{R}^d$4, removing MoE drops it by $h\in\mathbb{R}^d$5, the best number of sub-embeddings is $h\in\mathbb{R}^d$6, and the best number of experts is $h\in\mathbb{R}^d$7 (Pan et al., 29 Oct 2025).

FAME and its text-enhanced extension FAME+ place the MoE inside the final multi-head attention layer rather than in a feed-forward replacement. Each head is treated as a facet-specific predictor, and each head contains an MoE that disentangles varied user preferences within that facet through expert-specific query projections and a learnable router. FAME reports extensive experiments on four public sequential recommendation datasets and attributes its gains to multi-level gating: expert routing within heads and head-level gating across facets (Liu et al., 2024). FAME+ supplements this with a Text-Enhanced Facet-Aware Pre-training module built from a frozen pre-trained text encoder, a shared MLP, and $h\in\mathbb{R}^d$8 facet-specific projection heads trained with an alternating supervised contrastive objective before sequential fine-tuning (Liu et al., 18 Jan 2026). On Amazon-Beauty, Sports, Toys, and MovieLens-20m, FAME+ reports that it consistently achieves the best HR@K and NDCG@K, including Beauty NDCG@20 of $h\in\mathbb{R}^d$9 versus a best baseline of $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$0, Sports NDCG@20 of $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$1 versus $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$2, and ML-20m NDCG@20 of $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$3 versus $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$4 (Liu et al., 18 Jan 2026).

HyMoERec addresses a different weakness of sequential recommenders: the uniform Position-wise Feed-Forward Network. Its hybrid MoE combines a shared dense expert branch with a sparse expert branch, using top-$$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$5 routing over $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$6 experts with $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$7, a warm-up duration $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$8, and a load-balance regularizer weighted by $$H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})$$9 (Li et al., 9 Nov 2025). On MovieLens-1M and Amazon Beauty, with HR@5/10 and NDCG@5/10 as metrics, the paper reports HR@5 $k$0 and NDCG@5 $k$1 on ML-1M, compared with $k$2 and $k$3 for Mamba4Rec, and HR@5 $k$4 and NDCG@5 $k$5 on Beauty, compared with $k$6 and $k$7 (Li et al., 9 Nov 2025). The authors further state that removing the shared expert leads to unstable training and expert collapse, while fixing $k$8 rather than learning it degrades accuracy.

Across these recommendation models, SeqMoE is tied to three repeated themes: multi-faceted item structure, heterogeneous user preferences, and adaptive capacity allocation. The architectural differences are substantial, but the common objective is to avoid forcing a single representation pathway to model all sequential behaviors.

5. Sequential expert communication across depth and iterations

The most explicit sequential expert communication appears in CoE. "Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models" (Wang et al., 23 Jun 2025) processes tokens iteratively across a chain of experts inside a layer, with tokens allowed to re-evaluate and select different experts at each iteration. Under fixed compute on math reasoning tasks, CoE reports a reduction in validation loss from $k$9 to $y=\sum_{i=1}^N g_i \cdot f_i(h)$0 relative to a standard MoE. It also introduces what the paper calls a new scaling axis, iteration depth $y=\sum_{i=1}^N g_i \cdot f_i(h)$1, and reports that using $y=\sum_{i=1}^N g_i \cdot f_i(h)$2 iterations matches the performance of $y=\sum_{i=1}^N g_i \cdot f_i(h)$3 expert selections in width while reducing memory usage by $y=\sum_{i=1}^N g_i \cdot f_i(h)$4–$y=\sum_{i=1}^N g_i \cdot f_i(h)$5. The paper attributes the gains to the iterative residual structure and enhanced expert specialization empowered by iterative routing.

A related but distinct design is "Mixture of Raytraced Experts" (Perin et al., 16 Jul 2025), which dynamically selects sequences of experts and produces computational graphs of variable width and depth. Routing is carried out by repeatedly sampling from a candidate set through the Gumbel-softmax trick until an output node is selected, and the model is trained by unfolding the sequence in a manner compared to recurrent neural network training. The paper reports no explicit load-balancing mechanism, a reduction in training epochs of $y=\sum_{i=1}^N g_i \cdot f_i(h)$6 to $y=\sum_{i=1}^N g_i \cdot f_i(h)$7, and mean test accuracies of $y=\sum_{i=1}^N g_i \cdot f_i(h)$8 on MNIST, $y=\sum_{i=1}^N g_i \cdot f_i(h)$9 on Fashion-MNIST, and $k$0 on CIFAR-10 for SeqMoE, against $k$1, $k$2, and $k$3 for a top-$k$4 MoE baseline. The average number of experts used is $k$5 on MNIST, $k$6 on Fashion-MNIST, and $k$7 on CIFAR-10 (Perin et al., 16 Jul 2025).

Mammo-Mamba adapts sequential expert mixing to high-resolution medical imaging. In its deeper stages, experts are arranged in series: Stage 3 uses $k$8 experts in series, consisting of $k$9 SecMamba blocks followed by $k\cdot d$0 self-attention blocks, while Stage 4 uses $k\cdot d$1 experts, consisting of $k\cdot d$2 SecMamba blocks and $k\cdot d$3 attention blocks (Bayatmakou et al., 23 Jul 2025). After each expert, the gate chooses how much of the expert output to accept versus how much of the prior representation to retain. The model is trained end-to-end on benign-versus-malignant classification with cross-entropy, AdamW, label smoothing $k\cdot d$4, and no explicit expert dropout or auxiliary gating losses. The complexity analysis states $k\cdot d$5 for full self-attention, $k\cdot d$6 for a SecMamba layer, and $k\cdot d$7 for SeqMoE gating. In the reported ablation, dual-stream MambaVision without SeqMoE yields Accuracy $k\cdot d$8 and AUC $k\cdot d$9, whereas full Mammo-Mamba yields Accuracy $N\cdot d$0 and AUC $N\cdot d$1 (Bayatmakou et al., 23 Jul 2025).

Taken together, these papers define a branch of SeqMoE research in which sequentiality means ordered expert communication in depth or iteration. The emphasis is less on static expert assignment and more on whether later routing decisions can exploit intermediate computations produced by earlier experts.

6. Sequential inference and statistical mixture-of-experts

Long before recent neural SeqMoE work, sequential mixtures of experts appeared in online statistical inference. "Adaptive sequential Monte Carlo by means of mixture of experts" (Cornebise et al., 2011) approximates the optimal proposal kernel of a particle filter using a mixture of integrated curved exponential distributions with logistic weights. The mixtures are fitted by online-EM through minimization of the Kullback-Leibler divergence between auxiliary target and instrumental distributions, and each particle-filter iteration requires only a single optimization problem for the whole particle sample, yielding linear complexity. The paper reports that, in a bimodal Gaussian example, a single EM iteration raises the ESS from about $N\cdot d$2 to $N\cdot d$3 of $N\cdot d$4 and reduces the KL divergence by more than $N\cdot d$5; in the 2D ring-shaped Bessel toy, ESS doubles and negated Shannon entropy falls by more than $N\cdot d$6 after adaptation (Cornebise et al., 2011).

"Dynamic Mixture of Experts Models for Online Prediction" (Munezero et al., 2021) extends the finite mixture-of-experts model by allowing both component parameters and mixture weights to evolve in time according to random walks. Posterior inference is carried out by a sequential Monte Carlo algorithm with a tailored proposal distribution built from linear Bayes and EM ideas. In the software-fault-prediction application, the paper reports that a two-component dynamic Poisson mixture with discount factor $N\cdot d$7 gains over $N\cdot d$8 log-predictive-score points compared with a static mixture, and that ESS often remains at least $N\cdot d$9 of the particle count $T$00 (Munezero et al., 2021).

"Horseshoe Mixtures-of-Experts (HS-MoE)" (Polson et al., 14 Jan 2026) adds a Bayesian sparse-selection layer to this tradition by combining input-dependent gating with horseshoe global-local shrinkage. Its principal algorithmic contribution is a particle learning recursion that tracks only sufficient statistics while propagating the filter forward in time. The reported complexity is approximately $T$01 per time step, dominated by Student-$T$02 predictives and rank-one linear-algebra updates. The paper explicitly connects this sequential Bayesian MoE perspective to modern sparse MoE layers in LLMs, particularly under extreme sparsity constraints (Polson et al., 14 Jan 2026).

This statistical line of work uses the SeqMoE label in a different sense from neural expert-chaining models. The experts are probabilistic components, the gates are usually logistic or stick-breaking weights, and the core difficulty is online posterior updating rather than token-wise feature transformation.

7. Interpretation, misconceptions, and research directions

A persistent misconception is that SeqMoE denotes a single standardized neural module. The literature does not support that interpretation. CoE uses iteration-specific routers and inner residuals inside a layer; Mammo-Mamba uses a depth-wise two-way gate between identity carry-over and a current expert block; FAME uses per-head MoE together with head-level gating; HyMoERec combines a dense shared branch with sparse specialists; adaptive SMC and dynamic MoE models use sequential filtering rather than neural routing (Wang et al., 23 Jun 2025); (Bayatmakou et al., 23 Jul 2025); (Liu et al., 2024); (Li et al., 9 Nov 2025); (Cornebise et al., 2011); (Munezero et al., 2021). This suggests that comparisons across papers must first identify what form of sequentiality is actually being studied.

A second misconception is that SeqMoE necessarily implies top-$T$03 sparse routing with an auxiliary load-balancing loss. Some models do use explicit balancing devices: Fuxi-MME injects random noise into the gate to encourage expert load balancing, and HyMoERec adds a load-balance regularizer (Pan et al., 29 Oct 2025); (Li et al., 9 Nov 2025). Others do not: Mammo-Mamba states that no explicit expert dropout or auxiliary gating losses are used, and Mixture of Raytraced Experts states that no explicit load-balancing term is added (Bayatmakou et al., 23 Jul 2025); (Perin et al., 16 Jul 2025). Sequentiality, therefore, should not be conflated with a single regularization recipe.

The forward-looking directions stated in the literature are also heterogeneous. CoE argues that future SeqMoE research should explore richer intra-layer expert interactions beyond static, parallel gating (Wang et al., 23 Jun 2025). HyMoERec proposes more powerful routers and dynamic expert creation or elimination during training as possible extensions (Li et al., 9 Nov 2025). FAME+ states that the same machinery can extend to session-based user flows in e-commerce, multi-objective dialogue systems, hierarchical action planning, and time-series forecasting with multiple seasonal cycles (Liu et al., 18 Jan 2026). A plausible implication is that the next phase of SeqMoE research will be shaped less by a single dominant template than by continued specialization of expert-routing mechanisms to domain structure, computational budgets, and online-adaptation requirements.