---
title: Expandable Side MoE (XSMoE)
url: https://www.emergentmind.com/topics/expandable-side-mixture-of-experts-xsmoe
type: topic
---

# Expandable Side MoE (XSMoE)

Searching arXiv for the specified paper and related uses of “Expandable Side Mixture-of-Experts” to ground the article in current literature.
Expandable Side Mixture-of-Experts (XSMoE) is a framework in which lightweight, expandable expert modules are attached to frozen backbone models as side-tuning components, allowing continual adaptation without overwriting previously acquired knowledge. In its canonical form for multimodal streaming recommendation, XSMoE attaches expandable expert networks to frozen pretrained encoders such as ViT and BERT, uses routers to combine backbone and expert outputs, and applies utilization-based pruning to maintain compactness [2508.05993]. Subsequent work uses “XSMoE” more broadly as an architectural pattern for dynamically growing side adapters or side expert pools in continual retrieval and progressive MoE training [2507.09924] [2605.13247].

## 1. Problem setting and motivation

Streaming recommender systems (SRSs) are widely deployed in real-world applications, where user interests shift and new items arrive over time. In this setting, effectively capturing users’ latest preferences is challenging, because interactions reflecting recent interests are limited and new items often lack sufficient feedback. A common solution is to enrich item representations using multimodal encoders, for example BERT or ViT, to extract visual and textual features. The XSMoE formulation argues that these encoders are pretrained on general-purpose tasks, are not tailored to user preference modeling, and overlook the fact that user tastes toward modality-specific features such as visual styles and textual tones can also drift over time [2508.05993].

The resulting technical difficulties are twofold. First, fine-tuning large multimodal encoders in a streaming regime is costly. Second, continuous model updates risk forgetting long-term user preferences. XSMoE is designed around these constraints: it keeps the pretrained backbone frozen, learns new patterns through side experts, and expands those experts in response to evolving user feedback. This design targets both cold start and shifting preferences in multimodal features, rather than relying only on ID-based signals or repeated backbone retraining [2508.05993].

A common misconception is to view XSMoE as a standard MoE inserted directly into a fully trainable backbone. In the 2025 recommendation framework, the defining feature is instead side-tuning on top of frozen pretrained encoders, with growth localized to lightweight expert networks and routers rather than to the backbone itself [2508.05993].

## 2. Architectural organization

The reference architecture couples frozen visual and textual encoders with layerwise side-tuning and gated fusion. A schematic summary places a frozen ViT and a frozen BERT upstream of side-tuning blocks, each side-tuning block containing a set of experts \(E_{i,1}\dots E_{i,N}\) at layer \(i\) plus the frozen backbone output \(l_i\). The “G” boxes are routers, implemented as gating FFNs, and a pruning controller monitors expert utilizations and removes under-utilized experts periodically. After multimodal processing, the model forms \(e = \mathrm{FC}([e^v; e^t])\), feeds this representation into a SeqEncoder, and produces the recommendation score \(y_{u,i}\) [2508.05993].

The expandable aspect is defined operationally over time windows. During warm-up at time window \(0\), each side-tuning layer \(i\) has a single expert \(E_{i,1}\). At the beginning of each new time window \(s\), all existing experts \(E_{i,1\dots N_i}\) are frozen; a new FFN expert \(E_{i,N_i+1}\) is appended at each layer and initialized as the average of frozen experts; and the router \(G_i\) is expanded by adding one more output weight, described as one more column for the new expert. This yields an Expandable Mixture-of-Experts in which experts specialize on different time windows and never overwrite old parameters [2508.05993].

This architecture separates representational stability from adaptive plasticity. The frozen ViT and BERT preserve the pretrained multimodal backbone, while the side-tuning path absorbs streaming updates. A plausible implication is that the method treats expansion as a controlled memory mechanism: older experts retain historical preference structure, and newly appended experts encode recent behavioral drift.

## 3. Mathematical formulation

At layer \(i\), the frozen backbone encoder output is \(l_i \in \mathbb{R}^d\), the side-tuning input is \(h_{i-1} \in \mathbb{R}^d\), and the experts are \(E_{i,j}(h_{i-1})\) for \(j = 1 \dots N_i\). Router weights are defined by
\[
\alpha_i = \mathrm{Softmax}(W_i h_{i-1}) \in \mathbb{R}^{N_i+1},
\]
where \(\alpha_{i,0}\) corresponds to the backbone and \(\alpha_{i,j}\) to expert \(j\). The layer output is
\[
h_i = \alpha_{i,0} \cdot l_i + \sum_{j=1}^{N_i} \alpha_{i,j} \cdot E_{i,j}(h_{i-1}).
\]
Each expert is a small feed-forward network,
\[
E_{i,j}(h) = W_{\mathrm{up}}\, \mathrm{GELU}(W_{\mathrm{down}}\, h) + h.
\]
These equations make the side path an additive, router-weighted correction to the frozen backbone representation [2508.05993].

For recommendation, a user \(u\)’s sequence \([x_1 \dots x_L]\) is mapped to embeddings \([e_1 \dots e_L]\), the user vector is \(e_u = \mathrm{SeqEncoder}([e_1 \dots e_L])\), and the predicted score for item \(i\) is
\[
y_{u,i} = e_u \cdot e_i.
\]
Training uses In-Batch Debiased Cross-Entropy:
\[
D_{u,i} = \exp(y_{u,i} - \log p_i) + \sum_{j \in B \setminus I_u} \exp(y_{u,j} - \log p_j),
\]
\[
L_{CE} = -\sum_{u,i} \log \left(\frac{\exp(y_{u,i} - \log p_i)}{D_{u,i}}\right),
\]
where \(p_i\) is item popularity, \(I_u\) are items interacted by \(u\), and \(B\) is the batch [2508.05993].

The formulation combines three distinct mechanisms: frozen multimodal encoding, router-mediated interpolation between backbone and side experts, and a debiased ranking objective that explicitly corrects for item popularity. This suggests that XSMoE is not merely a PEFT wrapper around multimodal encoders; it is a continual recommendation system whose update path, ranking loss, and expansion policy are jointly specified.

## 4. Continual update, pruning, and forgetting mitigation

The update procedure is organized around time-ordered data \(D = [D_0 \dots D_T]\). The algorithm first precomputes and caches backbone outputs \(l_0 \dots l_M\), initializes side-tuning with one expert per layer, and performs warm-up on \(D_0\). For each subsequent time window \(s = 1 \dots T\), the framework splits \(D_s\) into train and validation sets, trains trainable experts and routers on \(D_s\) using \(L_{CE}\), evaluates on \(D_{s+1}\) with HR@10 and NDCG@10, computes expert utilizations for every layer, freezes all experts, appends a new expert and router column at each layer, and prunes underutilized experts [2508.05993].

Pruning is utilization-based. For expert \(j\) with output \(z_j = E_j(h)\) and router weight \(\alpha_j\), utilization is defined as
\[
r_j = \frac{\|\alpha_j z_j\|_2}{\sum_{i=1}^{N} \|\alpha_i z_i\|_2}.
\]
Experts with \(r_j < \tau\) are pruned, with one expert removed per layer per window. In effect, expansion and pruning operate together: growth accommodates newly observed preference patterns, while pruning constrains the accumulation of low-impact experts [2508.05993].

The framework’s catastrophic-forgetting mitigation strategy is explicit. New experts encode fresh patterns; old experts are frozen so past knowledge is never overwritten; and the router softly interpolates among experts, letting the model reuse old skills when appropriate. This is the central retention mechanism in the original XSMoE formulation, and it differs from rehearsal-based or full-model continual fine-tuning strategies by isolating new learning in appended side modules [2508.05993].

## 5. Computational profile and empirical behavior

The computational analysis uses \(M\) layers, input/output dimension \(d\), down-projection \(d'\), and \(N\) experts per layer after pruning, with \(N \ll d\). Total side-tuning parameters are
\[
M \cdot [N \cdot (2dd') + (dN + d)] \simeq O(M \cdot N \cdot d \cdot d').
\]
Trainable parameters, because only one new expert and router are active per layer, are
\[
M \cdot [2dd' + (dN + d)] \simeq O(M \cdot d \cdot d').
\]
Per-epoch training time includes forward through all experts and routers at \(O(M \cdot N \cdot d \cdot d')\) and backward plus update only for the active expert and router at \(O(M \cdot d \cdot d')\), yielding overall \(O(M \cdot N \cdot d \cdot d')\) per epoch. GPU memory per layer is \(O(N d d')\) for weights, \(O(d d')\) for gradients, and \(O(N d)\) for activations, giving \(O(M \cdot N \cdot d \cdot d')\) over \(M\) layers. Compared to full fine-tuning of ViT+BERT or other Adapter/LoRA PEFTs, XSMoE requires only \(O(M \cdot d \cdot d') \ll\) full-model parameters per update and does not store backbone activations [2508.05993].

The empirical evaluation uses three real-world datasets, time-split into 10 equal chunks: Amazon Home and Amazon Electronics from May 1996 to Oct 2018, and H&M Fashion from 2022. Baselines include ID-based SRS models SSRM, GAG, and GIUA-GNN; static multimodal models MMMLP, LGMRec, and IISAN; and each static multimodal baseline augmented with reservoir (RH) or KD regularization (KD). Evaluation uses HR@10 and NDCG@10 on chunks \(D_2 \dots D_9\) [2508.05993].

| Dataset | IISAN_RH (HR@10 / NDCG@10) | XSMoE (HR@10 / NDCG@10) |
|---|---:|---:|
| Amazon Home | 15.50 / 12.38 | **16.01 / 12.53** |
| Amazon Electronics | 13.66 / 10.39 | **13.88 / 10.48** |
| H&M Fashion | 24.94 / 20.21 | **25.44 / 20.38** |

XSMoE significantly outperforms all ID-based and static multimodal baselines with \(p < 0.01\). On Amazon Home, the efficiency comparison reports the following variants: NoFT (backbone only), with Total MP \(2.27\) MB, Trainable MP \(2.27\) MB, GPU Mem \(601\) MB, Epoch Time \(24.3\) s, and Total Time \(5\,694\) s; XSMoE (no pruning), with \(15.9\) MB total parameters, \(3.88\) MB trainable parameters, \(733\) MB GPU memory, \(24.1\) s epoch time, and \(3\,514\) s total time; and XSMoE with \(\tau = 0.10\), with \(8.32\) MB total parameters, \(3.82\) MB trainable parameters, \(616\) MB GPU memory, \(23.7\) s epoch time, and \(3\,246\) s total time. With moderate pruning \((\tau \approx 0.05\text{–}0.10)\), XSMoE converges in \(\sim 60\%\) of the time of NoFT, uses \(\sim \tfrac{1}{3}\) the GPU memory of full side-tuning, and retains top accuracy [2508.05993].

## 6. Generalization of the XSMoE pattern

Later work uses XSMoE as a more general continual-learning design principle. In MixLoRA-DSI, the architecture is described as a T5-based Differentiable Search Index frozen during continual updates except for small side adapters, and the expandable side module is realized as a mixture of LoRA experts inside selected Transformer decoder blocks. Each expert is a rank-\(r\) LoRA pair attached to \(W_{\mathrm{in}}\) and \(W_{\mathrm{out}}\), and new experts are introduced only when a layer-wise energy-based OOD criterion is triggered. The summary explicitly characterizes the method as realizing an XSMoE-style architecture in which small side LoRA experts are dynamically and selectively added only when energy-based OOD signals indicate that the existing adapters cannot cover newly arriving documents. The reported growth is sublinear: after indexing four increments on NQ320k, the method uses only 15 LoRA experts per layer rather than \(5\) layers \(\times\) \(1\) per corpus \(= 25\) under naive expansion [2507.09924].

EMO, by contrast, begins from the systems problem of sparse MoE training and then describes how to turn progressive expansion into an Expandable Side MoE. In that formulation, each MoE layer is replaced by two parallel expert pools, a main pool of \(E^{\mathrm{main}}\) experts and a side pool of \(E^{\mathrm{side}}\) experts, with separate routing distributions,
\[
p^{\mathrm{main}}(x)=\mathrm{softmax}(W^{\mathrm{main}}x+b^{\mathrm{main}}), \quad
p^{\mathrm{side}}(x)=\mathrm{softmax}(W^{\mathrm{side}}x+b^{\mathrm{side}}),
\]
and the combined layer output
\[
\mathrm{XSMoE}(x)=\sum_{i\in\mathcal K_{\mathrm{main}}(x)} p^{\mathrm{main}}_i\,E^{\mathrm{main}}_i(x)
+\sum_{j\in\mathcal K_{\mathrm{side}}(x)} p^{\mathrm{side}}_j\,E^{\mathrm{side}}_j(x).
\]
The scheduling rule treats the main and side pools as two pieces of expandable memory, interleaving expansions according to separate compute-optimal token schedules. This broadens XSMoE from a recommendation-specific PEFT framework into a generic recipe for growing “side” expert capacity alongside a shared main model [2605.13247].

Taken together, these later formulations indicate that “XSMoE” now denotes both a specific multimodal streaming recommendation framework and a broader expandable-side-expert pattern. The common invariant across these uses is architectural asymmetry: a large frozen or shared backbone remains stable, while small side experts provide the locus of adaptation.

## 7. Strengths, limitations, and future directions

The reported strengths of XSMoE are continual adaptation without catastrophic forgetting via expandable MoE and frozen experts, parameter-efficient updates because only side experts and routers are trained, dynamic model compactness via utilization-based pruning, and multimodal drift capture without retraining huge backbones [2508.05993].

The limitations are equally explicit. Expert expansion still increases model size, and threshold tuning for \(\tau\) affects trade-offs. ID embeddings are not yet integrated, so the original framework uses only multimodal features. Evaluation assumes all past items remain candidate and does not address item obsolescence. These constraints define the current scope of the method more precisely than broad descriptions of continual multimodal recommendation [2508.05993].

The future directions named for the framework are to integrate user/item ID embeddings alongside multimodal experts in streaming, extend to additional modalities such as audio and hierarchical metadata, explore adaptive gating priors or sparsity-inducing regularizers to further reduce experts, and conduct real-world A/B testing in e-commerce to handle item churn and cold-start more holistically [2508.05993]. A plausible implication, reinforced by MixLoRA-DSI and EMO, is that XSMoE has become a reusable continual-learning template for cases where backbone stability, parameter efficiency, and expandable specialization must be balanced within a single system.

Source: https://www.emergentmind.com/topics/expandable-side-mixture-of-experts-xsmoe