Expandable Side MoE (XSMoE)

• The paper shows that XSMoE delivers continual adaptation by attaching lightweight side experts to frozen backbones, preserving long-term knowledge while integrating new patterns.
• XSMoE dynamically appends expert modules and routers at each time window and employs utilization-based pruning to maintain model compactness.
• Empirical results demonstrate that XSMoE outperforms static baselines in HR@10 and NDCG@10 across real-world datasets, confirming its effectiveness in handling multimodal drift.

Searching arXiv for the specified paper and related uses of “Expandable Side Mixture-of-Experts” to ground the article in current literature. Expandable Side Mixture-of-Experts (XSMoE) is a framework in which lightweight, expandable expert modules are attached to frozen backbone models as side-tuning components, allowing continual adaptation without overwriting previously acquired knowledge. In its canonical form for multimodal streaming recommendation, XSMoE attaches expandable expert networks to frozen pretrained encoders such as ViT and BERT, uses routers to combine backbone and expert outputs, and applies utilization-based pruning to maintain compactness (Qu et al., 8 Aug 2025). Subsequent work uses “XSMoE” more broadly as an architectural pattern for dynamically growing side adapters or side expert pools in continual retrieval and progressive MoE training (Huynh et al., 14 Jul 2025, Jin et al., 13 May 2026).

1. Problem setting and motivation

Streaming recommender systems (SRSs) are widely deployed in real-world applications, where user interests shift and new items arrive over time. In this setting, effectively capturing users’ latest preferences is challenging, because interactions reflecting recent interests are limited and new items often lack sufficient feedback. A common solution is to enrich item representations using multimodal encoders, for example BERT or ViT, to extract visual and textual features. The XSMoE formulation argues that these encoders are pretrained on general-purpose tasks, are not tailored to user preference modeling, and overlook the fact that user tastes toward modality-specific features such as visual styles and textual tones can also drift over time (Qu et al., 8 Aug 2025).

The resulting technical difficulties are twofold. First, fine-tuning large multimodal encoders in a streaming regime is costly. Second, continuous model updates risk forgetting long-term user preferences. XSMoE is designed around these constraints: it keeps the pretrained backbone frozen, learns new patterns through side experts, and expands those experts in response to evolving user feedback. This design targets both cold start and shifting preferences in multimodal features, rather than relying only on ID-based signals or repeated backbone retraining (Qu et al., 8 Aug 2025).

A common misconception is to view XSMoE as a standard MoE inserted directly into a fully trainable backbone. In the 2025 recommendation framework, the defining feature is instead side-tuning on top of frozen pretrained encoders, with growth localized to lightweight expert networks and routers rather than to the backbone itself (Qu et al., 8 Aug 2025).

2. Architectural organization

The reference architecture couples frozen visual and textual encoders with layerwise side-tuning and gated fusion. A schematic summary places a frozen ViT and a frozen BERT upstream of side-tuning blocks, each side-tuning block containing a set of experts $E_{i,1}\dots E_{i,N}$ at layer $i$ plus the frozen backbone output $l_i$. The “G” boxes are routers, implemented as gating FFNs, and a pruning controller monitors expert utilizations and removes under-utilized experts periodically. After multimodal processing, the model forms $e = \mathrm{FC}([e^v; e^t])$, feeds this representation into a SeqEncoder, and produces the recommendation score $y_{u,i}$ (Qu et al., 8 Aug 2025).

The expandable aspect is defined operationally over time windows. During warm-up at time window $0$, each side-tuning layer $i$ has a single expert $E_{i,1}$. At the beginning of each new time window $s$, all existing experts $E_{i,1\dots N_i}$ are frozen; a new FFN expert $i$0 is appended at each layer and initialized as the average of frozen experts; and the router $i$1 is expanded by adding one more output weight, described as one more column for the new expert. This yields an Expandable Mixture-of-Experts in which experts specialize on different time windows and never overwrite old parameters (Qu et al., 8 Aug 2025).

This architecture separates representational stability from adaptive plasticity. The frozen ViT and BERT preserve the pretrained multimodal backbone, while the side-tuning path absorbs streaming updates. A plausible implication is that the method treats expansion as a controlled memory mechanism: older experts retain historical preference structure, and newly appended experts encode recent behavioral drift.

3. Mathematical formulation

At layer $i$2, the frozen backbone encoder output is $i$3, the side-tuning input is $i$4, and the experts are $i$5 for $i$6. Router weights are defined by

$i$7

where $i$8 corresponds to the backbone and $i$9 to expert $l_i$0. The layer output is

$l_i$1

Each expert is a small feed-forward network,

$l_i$2

These equations make the side path an additive, router-weighted correction to the frozen backbone representation (Qu et al., 8 Aug 2025).

For recommendation, a user $l_i$3’s sequence $l_i$4 is mapped to embeddings $l_i$5, the user vector is $l_i$6, and the predicted score for item $l_i$7 is

$l_i$8

Training uses In-Batch Debiased Cross-Entropy: $l_i$9

$e = \mathrm{FC}([e^v; e^t])$0

where $e = \mathrm{FC}([e^v; e^t])$1 is item popularity, $e = \mathrm{FC}([e^v; e^t])$2 are items interacted by $e = \mathrm{FC}([e^v; e^t])$3, and $e = \mathrm{FC}([e^v; e^t])$4 is the batch (Qu et al., 8 Aug 2025).

The formulation combines three distinct mechanisms: frozen multimodal encoding, router-mediated interpolation between backbone and side experts, and a debiased ranking objective that explicitly corrects for item popularity. This suggests that XSMoE is not merely a PEFT wrapper around multimodal encoders; it is a continual recommendation system whose update path, ranking loss, and expansion policy are jointly specified.

4. Continual update, pruning, and forgetting mitigation

The update procedure is organized around time-ordered data $e = \mathrm{FC}([e^v; e^t])$5. The algorithm first precomputes and caches backbone outputs $e = \mathrm{FC}([e^v; e^t])$6, initializes side-tuning with one expert per layer, and performs warm-up on $e = \mathrm{FC}([e^v; e^t])$7. For each subsequent time window $e = \mathrm{FC}([e^v; e^t])$8, the framework splits $e = \mathrm{FC}([e^v; e^t])$9 into train and validation sets, trains trainable experts and routers on $y_{u,i}$0 using $y_{u,i}$1, evaluates on $y_{u,i}$2 with HR@10 and NDCG@10, computes expert utilizations for every layer, freezes all experts, appends a new expert and router column at each layer, and prunes underutilized experts (Qu et al., 8 Aug 2025).

Pruning is utilization-based. For expert $y_{u,i}$3 with output $y_{u,i}$4 and router weight $y_{u,i}$5, utilization is defined as

$y_{u,i}$6

Experts with $y_{u,i}$7 are pruned, with one expert removed per layer per window. In effect, expansion and pruning operate together: growth accommodates newly observed preference patterns, while pruning constrains the accumulation of low-impact experts (Qu et al., 8 Aug 2025).

The framework’s catastrophic-forgetting mitigation strategy is explicit. New experts encode fresh patterns; old experts are frozen so past knowledge is never overwritten; and the router softly interpolates among experts, letting the model reuse old skills when appropriate. This is the central retention mechanism in the original XSMoE formulation, and it differs from rehearsal-based or full-model continual fine-tuning strategies by isolating new learning in appended side modules (Qu et al., 8 Aug 2025).

5. Computational profile and empirical behavior

The computational analysis uses $y_{u,i}$8 layers, input/output dimension $y_{u,i}$9, down-projection $0$0, and $0$1 experts per layer after pruning, with $0$2. Total side-tuning parameters are

$0$3

Trainable parameters, because only one new expert and router are active per layer, are

$0$4

Per-epoch training time includes forward through all experts and routers at $0$5 and backward plus update only for the active expert and router at $0$6, yielding overall $0$7 per epoch. GPU memory per layer is $0$8 for weights, $0$9 for gradients, and $i$0 for activations, giving $i$1 over $i$2 layers. Compared to full fine-tuning of ViT+BERT or other Adapter/LoRA PEFTs, XSMoE requires only $i$3 full-model parameters per update and does not store backbone activations (Qu et al., 8 Aug 2025).

The empirical evaluation uses three real-world datasets, time-split into 10 equal chunks: Amazon Home and Amazon Electronics from May 1996 to Oct 2018, and H&M Fashion from 2022. Baselines include ID-based SRS models SSRM, GAG, and GIUA-GNN; static multimodal models MMMLP, LGMRec, and IISAN; and each static multimodal baseline augmented with reservoir (RH) or KD regularization (KD). Evaluation uses HR@10 and NDCG@10 on chunks $i$4 (Qu et al., 8 Aug 2025).

Dataset	IISAN_RH (HR@10 / NDCG@10)	XSMoE (HR@10 / NDCG@10)
Amazon Home	15.50 / 12.38	16.01 / 12.53
Amazon Electronics	13.66 / 10.39	13.88 / 10.48
H&M Fashion	24.94 / 20.21	25.44 / 20.38

XSMoE significantly outperforms all ID-based and static multimodal baselines with $i$5. On Amazon Home, the efficiency comparison reports the following variants: NoFT (backbone only), with Total MP $i$6 MB, Trainable MP $i$7 MB, GPU Mem $i$8 MB, Epoch Time $i$9 s, and Total Time $E_{i,1}$0 s; XSMoE (no pruning), with $E_{i,1}$1 MB total parameters, $E_{i,1}$2 MB trainable parameters, $E_{i,1}$3 MB GPU memory, $E_{i,1}$4 s epoch time, and $E_{i,1}$5 s total time; and XSMoE with $E_{i,1}$6, with $E_{i,1}$7 MB total parameters, $E_{i,1}$8 MB trainable parameters, $E_{i,1}$9 MB GPU memory, $s$0 s epoch time, and $s$1 s total time. With moderate pruning $s$2, XSMoE converges in $s$3 of the time of NoFT, uses $s$4 the GPU memory of full side-tuning, and retains top accuracy (Qu et al., 8 Aug 2025).

6. Generalization of the XSMoE pattern

Later work uses XSMoE as a more general continual-learning design principle. In MixLoRA-DSI, the architecture is described as a T5-based Differentiable Search Index frozen during continual updates except for small side adapters, and the expandable side module is realized as a mixture of LoRA experts inside selected Transformer decoder blocks. Each expert is a rank-$s$5 LoRA pair attached to $s$6 and $s$7, and new experts are introduced only when a layer-wise energy-based OOD criterion is triggered. The summary explicitly characterizes the method as realizing an XSMoE-style architecture in which small side LoRA experts are dynamically and selectively added only when energy-based OOD signals indicate that the existing adapters cannot cover newly arriving documents. The reported growth is sublinear: after indexing four increments on NQ320k, the method uses only 15 LoRA experts per layer rather than $s$8 layers $s$9 $E_{i,1\dots N_i}$0 per corpus $E_{i,1\dots N_i}$1 under naive expansion (Huynh et al., 14 Jul 2025).

EMO, by contrast, begins from the systems problem of sparse MoE training and then describes how to turn progressive expansion into an Expandable Side MoE. In that formulation, each MoE layer is replaced by two parallel expert pools, a main pool of $E_{i,1\dots N_i}$2 experts and a side pool of $E_{i,1\dots N_i}$3 experts, with separate routing distributions,

$E_{i,1\dots N_i}$4

and the combined layer output

$E_{i,1\dots N_i}$5

The scheduling rule treats the main and side pools as two pieces of expandable memory, interleaving expansions according to separate compute-optimal token schedules. This broadens XSMoE from a recommendation-specific PEFT framework into a generic recipe for growing “side” expert capacity alongside a shared main model (Jin et al., 13 May 2026).

Taken together, these later formulations indicate that “XSMoE” now denotes both a specific multimodal streaming recommendation framework and a broader expandable-side-expert pattern. The common invariant across these uses is architectural asymmetry: a large frozen or shared backbone remains stable, while small side experts provide the locus of adaptation.

7. Strengths, limitations, and future directions

The reported strengths of XSMoE are continual adaptation without catastrophic forgetting via expandable MoE and frozen experts, parameter-efficient updates because only side experts and routers are trained, dynamic model compactness via utilization-based pruning, and multimodal drift capture without retraining huge backbones (Qu et al., 8 Aug 2025).

The limitations are equally explicit. Expert expansion still increases model size, and threshold tuning for $E_{i,1\dots N_i}$6 affects trade-offs. ID embeddings are not yet integrated, so the original framework uses only multimodal features. Evaluation assumes all past items remain candidate and does not address item obsolescence. These constraints define the current scope of the method more precisely than broad descriptions of continual multimodal recommendation (Qu et al., 8 Aug 2025).

The future directions named for the framework are to integrate user/item ID embeddings alongside multimodal experts in streaming, extend to additional modalities such as audio and hierarchical metadata, explore adaptive gating priors or sparsity-inducing regularizers to further reduce experts, and conduct real-world A/B testing in e-commerce to handle item churn and cold-start more holistically (Qu et al., 8 Aug 2025). A plausible implication, reinforced by MixLoRA-DSI and EMO, is that XSMoE has become a reusable continual-learning template for cases where backbone stability, parameter efficiency, and expandable specialization must be balanced within a single system.