MoRE: Mixture-of-Residual-Experts

• MoRE is a design pattern that separates a stable backbone from conditional residual experts, which add corrective refinements rather than replacing the core model output.
• It enables parameter-efficient fine-tuning and targeted corrections across various domains, including clinical sequence modeling, vision transformers, and humanoid control.
• Empirical results show improved accuracy and robustness, though increased complexity, expert routing challenges, and dependency on strong base models remain key limitations.

Mixture-of-Residual-Experts (MoRE) denotes a class of architectures in which a base model remains responsible for the main computation while a routed collection of experts learns additive corrections rather than full replacements. In the cited literature, this residualization appears in several forms: output-level correction on top of a pretrained GRU in clinical sequence modeling, feature-space correction on top of a frozen humanoid controller, latent residual modulation of a locomotion policy, weight-space decomposition into an input-independent core plus input-dependent residuals in vision transformers, and hierarchical low-rank residual adapters for LLM fine-tuning (Lee et al., 2022, Jing et al., 27 Aug 2025, Wang et al., 10 Jun 2025, Wu et al., 2022, Zeng et al., 8 Apr 2025). The term is also nomenclaturally unstable: several papers use “MoRE” for Mixture of Rule Experts, Mixture-of-Reasoning-Experts, or Mixture of Robust Experts rather than Mixture-of-Residual-Experts (Koebler et al., 26 Mar 2025, Si et al., 2023, Cheng et al., 2021).

1. Definition and terminological scope

Within residual-expert formulations, the defining move is to separate a stable backbone from a conditional correction mechanism. Standard MoE combines expert outputs directly, as in the classical form $y=\sum_{i=1}^N g_i(x)E_i(x)$, or, in control, $a=\sum_k g_k(x)f_k(x)$ (Wang et al., 10 Jun 2025, Jing et al., 27 Aug 2025). MoRE changes the role of the experts: the backbone first computes a base representation or action, and the experts contribute an additive residual that refines that result.

This distinction is explicit in multiple papers. In FARM, the frozen base Transformer produces $z_{\text{base}}$, the MoE produces $\Delta z$, and the final representation is $z=z_{\text{base}}+\Delta z$ before decoding to actions (Jing et al., 27 Aug 2025). In the humanoid gait paper, the pretrained policy yields $\bm{z}_t^o$, the expert mixture yields $\bm{z}_t'$, and the latent sent to the actor head is $\bm{z}_t=\bm{z}_t^o+\bm{z}_t'$ (Wang et al., 10 Jun 2025). In clinical sequence modeling, the base GRU prediction $\bm{o}_{base}$ is refined by an MoE output $\bm{o}_{moe}$, giving $a=\sum_k g_k(x)f_k(x)$0 (Lee et al., 2022). In graph learning, ADaMoRE explicitly defines an enhanced representation as backbone output plus a mixture of residual experts (Chu et al., 24 Oct 2025).

A common misconception is that the acronym “MoRE” always names a residual-expert method. That is incorrect in the recent literature. “MoRE-LLM” uses MoRE to mean Mixture of Rule Experts and explicitly states that there is no notion of residual experts in the method (Koebler et al., 26 Mar 2025). “Getting MoRE out of Mixture of LLM Reasoning Experts” uses MoRE for Mixture-of-Reasoning-Experts (Si et al., 2023). “Mixture of Robust Experts (MoRE)” uses the acronym for robustness to heterogeneous perturbations (Cheng et al., 2021). “MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts” is a 3D MoE model and explicitly notes that it is not a special Mixture-of-Residual-Experts variant (Gao et al., 31 Oct 2025).

2. Canonical formulation and relation to standard MoE

Across the cited residual-expert models, a common pattern is a decomposition of the form

$a=\sum_k g_k(x)f_k(x)$1

where $a=\sum_k g_k(x)f_k(x)$2 is a strong base model, $a=\sum_k g_k(x)f_k(x)$3 are residual experts, and $a=\sum_k g_k(x)f_k(x)$4 are routing weights. This synthesis matches the explicit formulas in FARM, gait MoRE, clinical R-MoE, and ADaMoRE, even though each paper places the residual at a different level of the computation (Jing et al., 27 Aug 2025, Wang et al., 10 Jun 2025, Lee et al., 2022, Chu et al., 24 Oct 2025).

The main technical consequence is specialization under a constrained correction budget. In standard MoE, each expert must model a full mapping. In MoRE, experts only learn the part that the base model misses. FARM states this explicitly: the frozen FC controller handles “routine” low-dynamic behaviors, and experts learn only the $a=\sum_k g_k(x)f_k(x)$5 needed to fix high-dynamic errors (Jing et al., 27 Aug 2025). The gait paper makes the same design choice in latent space: the existing actor head is preserved, and the residual latent modulates a terrain-capable base locomotion policy rather than replacing it (Wang et al., 10 Jun 2025).

Residualization also appears in parameter space. The vision-transformer paper “Residual Mixture of Experts” factorizes MoE weights into an input-independent core and an input-dependent residual, writing $a=\sum_k g_k(x)f_k(x)$6 (Wu et al., 2022). S’MoRE uses low-rank residual experts arranged hierarchically; tokens are routed through sub-trees of residuals, and the final adapter behaves as a residual added to the frozen transformer module, with parameter count and FLOPs essentially equal to a single high-rank LoRA (Zeng et al., 8 Apr 2025).

This suggests that “residual” in MoRE is not tied to one particular tensor location. The residual can be applied in output space, feature space, latent space, or weight space, so long as the experts operate as corrections to a pre-existing computation rather than as standalone full predictors.

3. Architectural realizations across domains

Paper and domain	Base component	Residual expert form
Clinical EHR (Lee et al., 2022)	Pretrained population-wide GRU	GRU experts produce $a=\sum_k g_k(x)f_k(x)$7 added to $a=\sum_k g_k(x)f_k(x)$8
Vision transformers (Wu et al., 2022)	Non-MoE ViT MLP weights as core	Experts initialized as core plus small noise; weights viewed as core + residual
FARM humanoid control (Jing et al., 27 Aug 2025)	Frozen FC input MLP + Transformer	Transformer adapters output feature residual $a=\sum_k g_k(x)f_k(x)$9
Humanoid gait learning (Wang et al., 10 Jun 2025)	Pretrained terrain locomotion policy	MLP experts output latent residual $z_{\text{base}}$0
LLM fine-tuning S’MoRE (Zeng et al., 8 Apr 2025)	Frozen transformer with LoRA-like adaptation point	Hierarchical low-rank residuals routed as sub-trees
Graph ADaMoRE (Chu et al., 24 Oct 2025)	Sparse MoE backbone encoder	Dense heterogeneous residual GNN experts added to backbone output

The architectural spread is important because it shows that MoRE is better understood as a design principle than as a single blueprint. In clinical prediction, the residual experts are full GRUs attached to a pretrained base GRU, with a GRU-based gating network producing softmax weights at each time step (Lee et al., 2022). In FARM, the residual experts are ControlNet-style Transformer adapters copied from the FC Transformer and followed by zero-initialized projection layers, so the residual path does not affect the initial behavior (Jing et al., 27 Aug 2025). In terrain-capable humanoid locomotion, each expert is a small MLP operating on actor features and gait commands, and the residual is added in policy latent space rather than in action space (Wang et al., 10 Jun 2025).

The parameter-efficient LLM setting pushes the idea further. S’MoRE does not simply maintain a flat bank of residual adapters; it organizes residuals of varying orders in multiple layers and interprets their propagation as a special type of GNN. Under similar parameter budget, it improves the “structural flexibility” of traditional MoE or Mixture-of-LoRA by exponential order (Zeng et al., 8 Apr 2025). ADaMoRE makes an analogous move on graphs, where a stable backbone handles most of the structure and residual experts contribute heterogeneous corrections trained under an unsupervised objective (Chu et al., 24 Oct 2025).

A separate but related control formulation appears in MRMEL, where a mixture of nominal policies is combined with a shared additive residual,

$z_{\text{base}}$1

so the residual does not belong to one expert but augments the routed nominal mixture (Jayawardana et al., 14 Jul 2025). This is still a residual-expert pattern, but the expert pool consists of nominal controllers rather than neural adapters.

4. Routing, specialization, and optimization

Routing is central to MoRE because it determines when residual capacity should be activated. In FARM, the router takes concatenated current and reference pose embeddings and outputs soft weights over two branches via a two-layer MLP with hidden size 256 and ReLU activation (Jing et al., 27 Aug 2025). A Speed-Aware Router supervises this gate using labels derived from average joint velocity, and Dynamic Expert-Assignment converts router probabilities into both the number of active experts and their weights. The reported implementation uses $z_{\text{base}}$2 residual experts, and low-dynamic segments often reduce to $z_{\text{base}}$3 or $z_{\text{base}}$4, effectively falling back to the base FC controller (Jing et al., 27 Aug 2025).

In the gait-learning MoRE, routing is conditioned jointly on actor features $z_{\text{base}}$5 and the one-hot gait command $z_{\text{base}}$6. The gate MLP produces logits $z_{\text{base}}$7, softmax weights mix the experts, and the resulting latent residual is added to the stage-1 locomotion latent (Wang et al., 10 Jun 2025). The paper experiments with 2, 3, and 4 experts and reports that the final model uses 3 experts aligned with walk–run, high-knees, and squat gait families (Wang et al., 10 Jun 2025).

Several papers treat routing as a specialization problem requiring explicit regularization. ADaMoRE combines structurally-aware gating, top-$z_{\text{base}}$8 sparse routing in the backbone, and a CKA-based diversity regularizer to enforce functional specialization among experts (Chu et al., 24 Oct 2025). S’MoRE provides sparse noisy top-$z_{\text{base}}$9, sparse Switch-Transformer gating, and dense gating, together with load-balancing losses when sparse routing is used (Zeng et al., 8 Apr 2025). By contrast, FARM notes that there is no explicit load-balancing regularizer; specialization is encouraged by speed supervision and by the capacity-scaling behavior of DEA (Jing et al., 27 Aug 2025).

Initialization and training schedules are equally important. FARM freezes the FC input MLP and Transformer, updates only the output MLP, and attaches the residual MoE module so that the initial behavior matches the pretrained controller exactly (Jing et al., 27 Aug 2025). The vision-transformer RMoE paper initializes each expert from the original MLP weights plus small noise and uses a StopGrad output-alignment trick during intermediate finetuning to avoid the output shrinkage that arises when a dense MLP is naively replaced by a sparse MoE layer (Wu et al., 2022). Clinical R-MoE is trained in two stages: first the base GRU, then the residual experts and gate with the base frozen, using a smaller learning rate and much larger weight decay for the MoE stage (Lee et al., 2022). S’MoRE likewise starts from a frozen base model and trains only residual adaptation parameters and routers (Zeng et al., 8 Apr 2025).

5. Empirical behavior and documented advantages

The strongest recurring empirical claim is that residual experts preserve the competence of a strong backbone while extending performance into failure regimes. On the High-Dynamic Humanoid Motion benchmark, FARM reduces the tracking failure rate by $\Delta z$0 and lowers global MPJPE by $\Delta z$1 relative to the FC baseline, while preserving near-perfect performance on low-dynamic motions; the full configuration reaches $\Delta z$2 success on HDHM versus $\Delta z$3 for FC, and on AMASS-test both FC and FARM retain $\Delta z$4 success (Jing et al., 27 Aug 2025). The ablations are especially diagnostic: “FA only, No MoE” shows essentially no improvement, while “FA + Res-MoE” improves HDHM and keeps AMASS-test degradation small, unlike “FA + Full-MoE,” which harms AMASS more substantially (Jing et al., 27 Aug 2025).

In clinical sequence modeling, the residual formulation also outperforms both a single RNN and a non-residual MoE. The paper reports a $\Delta z$5 gain on AUPRC compared to a single RNN prediction, with the best R-MoE reaching 31.6883 AUPRC versus 30.4383 for the GRU baseline (Lee et al., 2022). The residual-vs-non-residual ablation shows an average improvement of $\Delta z$6 AUPRC over the corresponding non-residual MoE across expert counts, indicating that the gain is not merely due to having more experts (Lee et al., 2022).

The vision-transformer RMoE paper emphasizes efficiency as much as accuracy. It reports comparable results with upper-bound full-MoE training while saving over $\Delta z$7 training cost, and, compared with state-of-the-art non-MoE transformers such as Swin-T, CvT-13, and Swin-L, reports $\Delta z$8 mIoU gain on ADE20K segmentation and $\Delta z$9 AP gain on MS-COCO object detection with less than $z=z_{\text{base}}+\Delta z$0 additional training cost (Wu et al., 2022). This is a distinct advantage of residual-expert formulations: the expensive part is learning the shared core, while expert specialization can often be obtained by short finetuning.

In locomotion on complex terrain, the residual placement matters. The gait paper reports that action-space residuals are unstable and fail to converge, whereas latent residuals are stable and high-performing; 3 experts give the best learning curve, and training from scratch without the base policy fails completely (Wang et al., 10 Jun 2025). On hard-gap terrain, base locomotion has success 0.893 and distance 12.465 m, while MoRE walk-run reaches 0.933 and 13.041 m; on hard steps, base locomotion has success 0.663 and distance 9.154 m, while MoRE high-knees reaches 0.947 and 13.247 m (Wang et al., 10 Jun 2025).

Graph learning and LLM adaptation extend the same pattern beyond control. ADaMoRE reports best performance on all 10 unsupervised node classification datasets and all 6 few-shot tasks, with ablations showing that removing residual experts or the diversity regularizer degrades performance (Chu et al., 24 Oct 2025). S’MoRE reports that, under similar parameter budget, its structural organization of low-rank residuals improves structural flexibility by exponential order and achieves superior fine-tuning performance relative to flat Mixture-of-LoRA baselines (Zeng et al., 8 Apr 2025). Taken together, these results suggest that residualization is most valuable when the base model already captures the dominant structure and the remaining error is heterogeneous, sparse, or regime-dependent.

6. Limitations, ambiguities, and research directions

Several limitations recur across the cited MoRE literature. Complexity is the most obvious. Even modest residual-expert systems are more complicated than their dense baselines. FARM notes that, even with only 2 experts, routing must be carefully supervised to avoid expert collapse, and performance is sensitive to both frame acceleration and expert count (Jing et al., 27 Aug 2025). The gait paper likewise reports that too few or too many experts are suboptimal, and that more experts can lead to over-fragmentation (Wang et al., 10 Jun 2025). ADaMoRE notes computational overhead from multiple channels and expert pools, as well as dependence on a sufficiently strong backbone for its sample-complexity argument (Chu et al., 24 Oct 2025). S’MoRE adds routing complexity and a larger hyperparameter surface, including depth, fanout, rank, and router design (Zeng et al., 8 Apr 2025).

A second limitation is dependence on the quality of the base model or prior. Clinical R-MoE assumes a strong pretrained GRU; the gait paper shows that training from scratch without the base policy fails completely; RMoE for vision transformers assumes access to a well-trained non-MoE upstream checkpoint; MRMEL assumes a useful pool of nominal policies and explicitly notes that the current framework is limited to continuous control (Lee et al., 2022, Wang et al., 10 Jun 2025, Wu et al., 2022, Jayawardana et al., 14 Jul 2025). This suggests that MoRE is most natural as an adaptation architecture rather than as a first-principles substitute for all dense models.

There are also domain-specific failure cases. FARM still fails on HDHM clips with subtle artifacts such as ground penetration, floating, and jitter (Jing et al., 27 Aug 2025). Humanoid gait learning remains constrained by reference motion quality and by the difficulty of modeling long-horizon choreographed gait transitions (Wang et al., 10 Jun 2025). Graph MoRE depends on a finite expert pool and on the adequacy of complementary LPF/HPF-style backbones for the target graph family (Chu et al., 24 Oct 2025). S’MoRE has only limited depth scaling evidence beyond 3 layers (Zeng et al., 8 Apr 2025).

The most common future directions are consistent across domains. FARM explicitly raises more or hierarchical experts, different residual placements, integration with planners or contact schedulers, diffusion-based priors, and real humanoid deployment (Jing et al., 27 Aug 2025). The gait paper suggests extension to more challenging and lifelike gaits and smoother transitions (Wang et al., 10 Jun 2025). ADaMoRE points to task-conditioned experts, dynamic backbones, online adaptation, and hierarchical MoRE stacks (Chu et al., 24 Oct 2025). S’MoRE motivates deeper hierarchies, richer routers, broader application domains, and scaling-law analyses that treat structure, not just expert count, as a capacity axis (Zeng et al., 8 Apr 2025).

A final ambiguity is lexical rather than technical: “MoRE” is now an overloaded acronym on arXiv. For precision, residual-expert papers increasingly require contextual disambiguation by title, domain, or explicit equations. In that narrower sense, Mixture-of-Residual-Experts is best understood not as one fixed architecture, but as a recurring pattern: preserve a competent backbone, allocate conditional capacity only where the backbone is insufficient, and train experts to model residual structure rather than full behavior.