---
title: Sequential Mixture of Experts (SeqMoE)
url: https://www.emergentmind.com/topics/sequential-mixture-of-experts-seqmoe
type: topic
---

# Sequential Mixture of Experts (SeqMoE)

Sequential Mixture of Experts (SeqMoE) designates a family of mixture-of-experts formulations in which expert specialization is coupled to a sequential process rather than being exhausted by a single parallel routing event. Across the literature, the term is used for non-identical mechanisms: divide-and-conquer expert specialization for chemical reaction prediction, sparse expert routing inside sequential recommenders, iterative expert communication within a layer, depth-wise gating across successive blocks, and sequential Bayesian inference in time-evolving mixture models [2310.04674]; [2510.25285]; [2506.18945]; [1108.2836]. This suggests that SeqMoE is best understood as a research lineage of related designs rather than a single canonical architecture.

## 1. Conceptual scope and nomenclature

In conventional sparse MoE layers, experts operate independently in parallel, with a token-level output of the form \(y=\sum_{i=1}^N g_i \cdot E_i(x)\), where only a small number of gates are active. "Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models" [2506.18945] defines its departure from this pattern by replacing one-shot routing with a \(T\)-step sequential relay inside a layer, so that the hidden state is updated iteratively as \(x^{(t)} = x^{(t-1)} + \sum_{i=1}^N g_{t,i}\cdot E_i(x^{(t-1)})\), with a dedicated router at each iteration. In that formulation, sequentiality means that later experts consume the intermediate representation produced by earlier experts rather than a fixed initial input.

Other papers use the same label for different forms of ordered expert interaction. "Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography" [2507.17662] organizes experts in depth and applies a gate after each expert to mix the current expert output with the running representation from the previous layer. "Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts" [2510.25285], "Facet-Aware Multi-Head Mixture-of-Experts Model for Sequential Recommendation" [2411.01457], and "HyMoERec: Hybrid Mixture-of-Experts for Sequential Recommendation" [2511.06388] embed MoE mechanisms inside sequential recommendation backbones, where the underlying task is sequential even when the expert computation at a given block is sparse rather than iterative. By contrast, "Adaptive sequential Monte Carlo by means of mixture of experts" [1108.2836], "Dynamic Mixture of Experts Models for Online Prediction" [2109.11449], and "Horseshoe Mixtures-of-Experts (HS-MoE)" [2601.09043] use sequentiality in the statistical sense: inference proceeds online in time through particle filtering, particle learning, or online EM.

The resulting terminological landscape is heterogeneous. A SeqMoE system may therefore denote sequential expert communication, sequential routing in depth, MoE-enhanced sequential modeling, or sequential posterior updating. This is a central point for interpreting the literature.

## 2. Routing, gating, and expert composition

The most common SeqMoE mechanism remains input-dependent gating. In Fuxi-MME, the gating logits for an input vector \(h\in\mathbb{R}^d\) are \(H(h)=h\cdot W_g + \mathcal N(0,I)\cdot Softplus(h\cdot W_{noise})\), only the top-\(k\) entries are retained, and the MoE output is \(y=\sum_{i=1}^N g_i \cdot f_i(h)\) [2510.25285]. The added noise term is used to encourage expert load balancing, while sparse top-\(k\) activation keeps the compute cost proportional to \(k\cdot d\) rather than \(N\cdot d\).

Other SeqMoE models use hierarchical or multi-level routing. In FAME, each attention head is treated as a distinct facet, and each head contains its own \(N\)-expert MoE for query generation. Expert outputs are aggregated through a router \(\beta_{i,\cdot}^{(h)} = softmax([f_{i,(1)}^{(h)}|\dots|f_{i,(N)}^{(h)}]^\top W_{exp}^{(h)})\), giving \(f_i^{(h)}=\sum_{n=1}^N \beta_{i,(n)}^{(h)} f_{i,(n)}^{(h)}\); a second gate \(\hat g=softmax([F_t^{(1)}|\dots|F_t^{(H)}]^\top W_g+b_g)\) then weights the heads themselves [2411.01457]. The architecture is therefore a two-level mixture: first among experts within a facet, then among facets.

Hybrid formulations mix dense and sparse branches rather than only sparse experts. HyMoERec defines a shared dense expert \(f_{\text{shared}}\), a specialized sparse expert set \(\{f_1,\dots,f_E\}\), and an Adaptive Expert Fusion rule \( \mathbf y=\mathbf y_{\text{dense}}+\alpha\,\mathbf y_{\text{MoE}}\), with \(\alpha=\sigma(\alpha_{\text{param}})\cdot w(t)\) and a linear warm-up schedule \(w(t)=\min(1,\tfrac{t}{T_{\text{warmup}}})\) where \(T_{\text{warmup}}=500\) [2511.06388]. Mammo-Mamba uses an even smaller gating simplex: after each expert block, a two-layer gating MLP produces \([g_1,g_2]\) with \(g_1+g_2=1\), and the new representation is \(X^{(\ell)}=g_1 e_{\rm prev}+g_2 e_{\rm new}\) [2507.17662].

Statistical SeqMoE models use probabilistic experts rather than neural FFNs. In adaptive SMC, the proposal kernel is written as
$$
q_\theta(x_t\mid x_{t-1})=\sum_{j=1}^d w_j(x_{t-1};\beta)\,\rho(x_{t-1},x_t;\eta_j),
$$
where the \(w_j(x_{t-1};\beta)\) are logistic gating weights and each expert \(\rho(\cdot;\eta_j)\) belongs to a family of integrated curved exponential distributions [1108.2836]. Here the gate partitions the ancestor space, and the experts model local proposal structure rather than feature transformations.

## 3. Reaction prediction and rare plausible outcomes

"Beyond the Typical: Modeling Rare Plausible Patterns in Chemical Reactions by Leveraging Sequential Mixture-of-Experts" [2310.04674] applies the SeqMoE idea to reaction prediction, defined there as predicting the outcome of a reaction from given reactants. The paper’s abstract states that generative models such as Transformer and VAE have typically been used for this task, but that these likelihood-maximization models overlook the inherent stochastic nature of chemical reactions, including the multiple ways electrons can be redistributed among atoms during the reaction process. When similar reactants can follow different electron redistribution patterns, the most common outcomes are typically predicted while less frequent but potentially crucial reaction patterns are neglected.

The proposed remedy is to organize the mapping space between reactants and electron redistribution patterns in a divide-and-conquer manner. Multiple expert models are trained, each specializing in a type of electron redistribution pattern in reaction. During inference, a dropout strategy is applied to each expert to improve electron redistribution diversity, and a ranking stage integrates predictions from multiple experts to produce the most plausible products. The reported experiments are conducted on the USPTO-MIT benchmark, described as the largest reaction prediction benchmark in the abstract, and the method is stated to outperform baselines [2310.04674].

The available description is precise about the divide-and-conquer motivation, expert specialization, expert dropout, and final ranking stage, but it does not provide gating equations, individual expert definitions, or sequential inference formulas for the SeqMoE mechanism itself [2310.04674]. This makes the reaction-prediction paper notable both for its application focus and for the relative lack of low-level architectural detail in the available summary.

## 4. SeqMoE in sequential recommendation

Sequential recommendation has been one of the most active application areas for SeqMoE-style designs. In Fuxi-MME, the conventional single embedding matrix is decomposed into several lower-dimensional embedding matrices \(E^{(1)},\dots,E^{(M)}\), each in \(\mathbb R^{|I|\times(d/M)}\). The Fuxi Block replaces its feed-forward network and gating projection with a sparse MoE layer containing \(N\) experts, and only the top-\(k\) experts are activated for a token [2510.25285]. The paper reports results on Amazon-Books, Amazon-Beauty, and Yelp using HR@10/50 and NDCG@10/50, and states that Fuxi-MME improves NDCG@10 on Amazon-Books by \(\approx 5.4\%\) over Fuxi-\(\alpha\) and by \(\sim 7\)–\(15\%\) over other baselines. Its ablations report that removing multi-embedding drops NDCG@10 by \(\sim 4\%\), removing MoE drops it by \(\sim 2\%\), the best number of sub-embeddings is \(M=4\), and the best number of experts is \(N=4\) [2510.25285].

FAME and its text-enhanced extension FAME+ place the MoE inside the final multi-head attention layer rather than in a feed-forward replacement. Each head is treated as a facet-specific predictor, and each head contains an MoE that disentangles varied user preferences within that facet through expert-specific query projections and a learnable router. FAME reports extensive experiments on four public sequential recommendation datasets and attributes its gains to multi-level gating: expert routing within heads and head-level gating across facets [2411.01457]. FAME+ supplements this with a Text-Enhanced Facet-Aware Pre-training module built from a frozen pre-trained text encoder, a shared MLP, and \(H\) facet-specific projection heads trained with an alternating supervised contrastive objective before sequential fine-tuning [2601.12301]. On Amazon-Beauty, Sports, Toys, and MovieLens-20m, FAME+ reports that it consistently achieves the best HR@K and NDCG@K, including Beauty NDCG@20 of \(0.0709\) versus a best baseline of \(0.0646\), Sports NDCG@20 of \(0.0418\) versus \(0.0370\), and ML-20m NDCG@20 of \(0.1608\) versus \(0.1450\) [2601.12301].

HyMoERec addresses a different weakness of sequential recommenders: the uniform Position-wise Feed-Forward Network. Its hybrid MoE combines a shared dense expert branch with a sparse expert branch, using top-\(K\) routing over \(E=4\) experts with \(K=2\), a warm-up duration \(T_{\text{warmup}}=500\), and a load-balance regularizer weighted by \(\lambda_{\mathrm{lb}}=0.02\) [2511.06388]. On MovieLens-1M and Amazon Beauty, with HR@5/10 and NDCG@5/10 as metrics, the paper reports HR@5 \(=0.2157\) and NDCG@5 \(=0.1486\) on ML-1M, compared with \(0.2106\) and \(0.1447\) for Mamba4Rec, and HR@5 \(=0.0518\) and NDCG@5 \(=0.0365\) on Beauty, compared with \(0.0511\) and \(0.0362\) [2511.06388]. The authors further state that removing the shared expert leads to unstable training and expert collapse, while fixing \(\alpha\) rather than learning it degrades accuracy.

Across these recommendation models, SeqMoE is tied to three repeated themes: multi-faceted item structure, heterogeneous user preferences, and adaptive capacity allocation. The architectural differences are substantial, but the common objective is to avoid forcing a single representation pathway to model all sequential behaviors.

## 5. Sequential expert communication across depth and iterations

The most explicit sequential expert communication appears in CoE. "Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models" [2506.18945] processes tokens iteratively across a chain of experts inside a layer, with tokens allowed to re-evaluate and select different experts at each iteration. Under fixed compute on math reasoning tasks, CoE reports a reduction in validation loss from \(1.20\) to \(1.12\) relative to a standard MoE. It also introduces what the paper calls a new scaling axis, iteration depth \(T\), and reports that using \(2\times\) iterations matches the performance of \(3\times\) expert selections in width while reducing memory usage by \(17.6\%\)–\(42\%\). The paper attributes the gains to the iterative residual structure and enhanced expert specialization empowered by iterative routing.

A related but distinct design is "Mixture of Raytraced Experts" [2507.12419], which dynamically selects sequences of experts and produces computational graphs of variable width and depth. Routing is carried out by repeatedly sampling from a candidate set through the Gumbel-softmax trick until an output node is selected, and the model is trained by unfolding the sequence in a manner compared to recurrent neural network training. The paper reports no explicit load-balancing mechanism, a reduction in training epochs of \(10\%\) to \(40\%\), and mean test accuracies of \(96.3\pm0.1\) on MNIST, \(87.1\pm0.1\) on Fashion-MNIST, and \(60.4\pm1.2\) on CIFAR-10 for SeqMoE, against \(95.3\pm0.3\), \(85.1\pm0.4\), and \(55.7\pm0.9\) for a top-\(k\) MoE baseline. The average number of experts used is \(7.5\) on MNIST, \(9.4\) on Fashion-MNIST, and \(11.4\) on CIFAR-10 [2507.12419].

Mammo-Mamba adapts sequential expert mixing to high-resolution medical imaging. In its deeper stages, experts are arranged in series: Stage 3 uses \(10\) experts in series, consisting of \(5\) SecMamba blocks followed by \(5\) self-attention blocks, while Stage 4 uses \(5\) experts, consisting of \(3\) SecMamba blocks and \(2\) attention blocks [2507.17662]. After each expert, the gate chooses how much of the expert output to accept versus how much of the prior representation to retain. The model is trained end-to-end on benign-versus-malignant classification with cross-entropy, AdamW, label smoothing \(\epsilon=0.1\), and no explicit expert dropout or auxiliary gating losses. The complexity analysis states \(O(N^2d)\) for full self-attention, \(O(Nd)\) for a SecMamba layer, and \(O(d)\) for SeqMoE gating. In the reported ablation, dual-stream MambaVision without SeqMoE yields Accuracy \(=0.8322\) and AUC \(=0.8791\), whereas full Mammo-Mamba yields Accuracy \(=0.8792\) and AUC \(=0.9249\) [2507.17662].

Taken together, these papers define a branch of SeqMoE research in which sequentiality means ordered expert communication in depth or iteration. The emphasis is less on static expert assignment and more on whether later routing decisions can exploit intermediate computations produced by earlier experts.

## 6. Sequential inference and statistical mixture-of-experts

Long before recent neural SeqMoE work, sequential mixtures of experts appeared in online statistical inference. "Adaptive sequential Monte Carlo by means of mixture of experts" [1108.2836] approximates the optimal proposal kernel of a particle filter using a mixture of integrated curved exponential distributions with logistic weights. The mixtures are fitted by online-EM through minimization of the Kullback-Leibler divergence between auxiliary target and instrumental distributions, and each particle-filter iteration requires only a single optimization problem for the whole particle sample, yielding linear complexity. The paper reports that, in a bimodal Gaussian example, a single EM iteration raises the ESS from about \(20\%\) to \(40\%\) of \(N\) and reduces the KL divergence by more than \(50\%\); in the 2D ring-shaped Bessel toy, ESS doubles and negated Shannon entropy falls by more than \(60\%\) after adaptation [1108.2836].

"Dynamic Mixture of Experts Models for Online Prediction" [2109.11449] extends the finite mixture-of-experts model by allowing both component parameters and mixture weights to evolve in time according to random walks. Posterior inference is carried out by a sequential Monte Carlo algorithm with a tailored proposal distribution built from linear Bayes and EM ideas. In the software-fault-prediction application, the paper reports that a two-component dynamic Poisson mixture with discount factor \(\alpha=0.5\) gains over \(100\) log-predictive-score points compared with a static mixture, and that ESS often remains at least \(50\%\) of the particle count \(M\) [2109.11449].

"Horseshoe Mixtures-of-Experts (HS-MoE)" [2601.09043] adds a Bayesian sparse-selection layer to this tradition by combining input-dependent gating with horseshoe global-local shrinkage. Its principal algorithmic contribution is a particle learning recursion that tracks only sufficient statistics while propagating the filter forward in time. The reported complexity is approximately \(O(NKd^2)\) per time step, dominated by Student-\(t\) predictives and rank-one linear-algebra updates. The paper explicitly connects this sequential Bayesian MoE perspective to modern sparse MoE layers in large language models, particularly under extreme sparsity constraints [2601.09043].

This statistical line of work uses the SeqMoE label in a different sense from neural expert-chaining models. The experts are probabilistic components, the gates are usually logistic or stick-breaking weights, and the core difficulty is online posterior updating rather than token-wise feature transformation.

## 7. Interpretation, misconceptions, and research directions

A persistent misconception is that SeqMoE denotes a single standardized neural module. The literature does not support that interpretation. CoE uses iteration-specific routers and inner residuals inside a layer; Mammo-Mamba uses a depth-wise two-way gate between identity carry-over and a current expert block; FAME uses per-head MoE together with head-level gating; HyMoERec combines a dense shared branch with sparse specialists; adaptive SMC and dynamic MoE models use sequential filtering rather than neural routing [2506.18945]; [2507.17662]; [2411.01457]; [2511.06388]; [1108.2836]; [2109.11449]. This suggests that comparisons across papers must first identify what form of sequentiality is actually being studied.

A second misconception is that SeqMoE necessarily implies top-\(k\) sparse routing with an auxiliary load-balancing loss. Some models do use explicit balancing devices: Fuxi-MME injects random noise into the gate to encourage expert load balancing, and HyMoERec adds a load-balance regularizer [2510.25285]; [2511.06388]. Others do not: Mammo-Mamba states that no explicit expert dropout or auxiliary gating losses are used, and Mixture of Raytraced Experts states that no explicit load-balancing term is added [2507.17662]; [2507.12419]. Sequentiality, therefore, should not be conflated with a single regularization recipe.

The forward-looking directions stated in the literature are also heterogeneous. CoE argues that future SeqMoE research should explore richer intra-layer expert interactions beyond static, parallel gating [2506.18945]. HyMoERec proposes more powerful routers and dynamic expert creation or elimination during training as possible extensions [2511.06388]. FAME+ states that the same machinery can extend to session-based user flows in e-commerce, multi-objective dialogue systems, hierarchical action planning, and time-series forecasting with multiple seasonal cycles [2601.12301]. A plausible implication is that the next phase of SeqMoE research will be shaped less by a single dominant template than by continued specialization of expert-routing mechanisms to domain structure, computational budgets, and online-adaptation requirements.

Source: https://www.emergentmind.com/topics/sequential-mixture-of-experts-seqmoe