Mixture of DoRA Experts (MoDE)

• Mixture of DoRA Experts is an emerging design paradigm that augments a frozen backbone with multiple specialized DoRA-style PEFT modules using dynamic routing.
• The architecture leverages principles from modular domain experts, projection-level PEFT, and dyadic expert composition to support multi-domain and task adaptations.
• Current challenges include terminological ambiguity and non-standardized expert instantiation, emphasizing the critical role of expert routing in these systems.

“Mixture of DoRA Experts” is not presently a standardized arXiv term. In the cited literature, the acronym “MoDE” denotes several different systems—“Modular Domain Experts,” “Mixture-of-Distilled-Expert(s),” “Mixture of Dyadic Experts,” and “Mixture of Deformation Experts”—and none of them is a DoRA-based architecture in the strict sense (Schafhalter et al., 2024, Xie et al., 2024, Ning et al., 2024, Jin et al., 9 Jul 2026). As an Editor’s term, “Mixture of DoRA Experts” can therefore denote the broader design space in which a frozen backbone is augmented with multiple specialized DoRA-style PEFT modules and those modules are combined by routing, weighted composition, or staged modular training. The closest direct evidence comes not from a canonical DoRA-MoDE paper, but from adjacent work on modular domain experts, mixtures of LoRA-style experts, dynamic routing, and structured PEFT mixtures (Li et al., 17 Jun 2025, Zhuang et al., 30 Sep 2025).

1. Terminological status and acronym ambiguity

The first fact about “Mixture of DoRA Experts” is terminological: the literature does not presently use “MoDE” as a settled name for such a method. Several unrelated papers employ the same acronym for different constructions.

Acronym in paper	Meaning	Relation to DoRA-expert mixtures
MoDE (Schafhalter et al., 2024)	Modular Domain Experts	Transformer-layer experts parallel to a frozen backbone; no DoRA
MoDE (Xie et al., 2024)	Mixture-of-Distilled-Expert(s)	Standard MoE with mutual distillation; no DoRA
MoDE (Ning et al., 2024)	Mixture of Dyadic Experts	LoRA/MoE variant with routed rank-one components; no DoRA
MoDE (Jin et al., 9 Jul 2026)	Mixture of Deformation Experts	Dynamic Gaussian Splatting; unrelated to PEFT DoRA

This ambiguity matters because it changes what counts as evidence. “Scalable Multi-Domain Adaptation of LLMs using Modular Experts” explicitly states that its experts are stacks of transformer layers rather than DoRA-style weight decompositions, so it supports a DoRA-expert mixture only indirectly (Schafhalter et al., 2024). “MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts” is closer to the PEFT setting, but it is a LoRA-based method that shares down-projection matrices and routes among dyadic rank-one components; it does not discuss DoRA (Ning et al., 2024). The adapter-mixing toolkit in “Flexible and Effective Mixing of LLMs into a Mixture of Domain Experts” is highly relevant operationally because it supports creating a mixture from trained models or adapters, but it likewise does not define a DoRA-specific MoDE (Lee et al., 2024).

A second terminological point is that DoRA itself appears in the supplied literature mainly as a comparator or a conceptual reference point, not as the basis of a mixture architecture. FourierMoE reports direct comparisons against DoRA in both multi-task reasoning and single-task settings, and DR-LoRA compares against a DoRA baseline when adapting pretrained MoE LLMs (Jiang et al., 2 Apr 2026, Deng et al., 8 Jan 2026). This places “Mixture of DoRA Experts” in an unusual position: the expert-mixing machinery is well represented in adjacent papers, while the DoRA-specific expert parameterization remains largely uninstantiated in the cited MoDE-named work.

2. Architectural patterns relevant to a DoRA-expert mixture

Adjacent work presents several distinct architectural motifs for composing experts around a frozen backbone. The most structurally explicit is Modular Domain Experts, where the pretrained LLM is partitioned into a sequence of MoDE blocks, and each block contains a subset of frozen backbone transformer layers together with one block from each domain expert. All branches process the same hidden state in parallel, and their outputs are merged by a learned weighted sum:

$$y = \alpha_{\text{bb}} f_{\text{bb}^i}(x) + \sum_{j=1}^{N} \alpha_{\text{e}^j} f_{\text{e}^{i,j}}(x),$$

with

$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$

In the reported 1.58B-parameter configuration, the backbone has 18 transformer layers divided into 3 MoDE blocks, and each expert contributes 2 transformer layers per block, for 6 layers per expert total (Schafhalter et al., 2024). This is not a PEFT construction, but it establishes the broader principle that independently trained, domain-specialized modules can be composed inside a frozen host model.

A second pattern is projection-level PEFT expertization. LoRA-Mixer replaces projection matrices in the model’s native computation path with dynamically routed LoRA experts, rather than adding detached side branches. The paper applies this “serial” integration to attention modules in transformer models and to the in, out, dt, and x projection layers in Falcon-Mamba, with the adapted projection output fed directly into downstream attention or state-space computation (Li et al., 17 Jun 2025). This is especially relevant because a DoRA expert mixture would most naturally occupy the same interfaces: selected linear projections in attention or FFN pathways rather than entire replacement layers.

A third pattern is sub-adapter factorization. Mixture of Dyadic Experts rewrites a LoRA update as a sum of rank-one dyadic products and routes among those atomic components. With shared down-projection columns $\mathbf{a}_j$ and expert-specific up vectors $\mathbf{b}_j^i$, the effective update is

$$\Delta \mathbf{W}_{\text{MoDE}(\mathbf{x})} = \sum_{i=1}^{m}\sum_{j=1}^{r} \mathcal{R}_j^i(\mathbf{x})\,(\mathbf{a}_j \otimes {\mathbf{b}_j^i}^{T}).$$

This makes the expert unit smaller than a whole adapter and shows that expert granularity can be moved from “one module per expert” to “one component per expert” (Ning et al., 2024). A plausible implication is that a DoRA-based mixture need not route whole DoRA modules only; it could also route finer subcomponents if the parameterization exposes them cleanly.

A fourth pattern is hierarchical decomposition by task and capability. DMoERM uses an outer sparse MoE over task categories and an inner dense MoE over capability dimensions, with each capability expert implemented as a LoRA specialization over a shared task-specific reward-model backbone (Quan, 2024). The architectural lesson is not limited to reward modeling: expert identity can correspond to task category, capability point, projection, frequency band, or low-rank component, rather than only to a broad “domain.”

Finally, FourierMoE shows that experts can be made structurally non-redundant by design. Instead of generic spatial-domain adapters, each expert specializes in a frequency band and learns conjugate-symmetric complex coefficients whose inverse DFT reconstructs a real-valued weight update (Jiang et al., 2 Apr 2026). This suggests that a DoRA-expert mixture may benefit from an explicit specialization axis, rather than from a pool of otherwise identical adapters.

3. Routing and composition mechanisms

Routing is the main technical axis along which adjacent methods differ. Modular Domain Experts uses a deliberately simple token-level gating function, $$g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x))$$, and composes backbone and expert outputs by dense weighted addition rather than by hard top-$k$ sparse dispatch (Schafhalter et al., 2024). This is a mixture in the literal sense of soft weighted combination, not a classical sparse pretraining MoE in which only a few experts execute.

Mixture of Dyadic Experts moves routing to a finer scale. Instead of assigning one weight per whole low-rank adapter, it uses a separate softmax router for each rank position:

$$\mathcal{R}_j(\mathbf{x}) = \operatorname{softmax}(\mathbf{x}\cdot \mathbf{W}_{\mathcal{R};j}),$$

so routing is simultaneously token-level and component-level (Ning et al., 2024). This changes the effective combinatorial space from choosing among a small number of full adapters to constructing an input-conditioned adapter from many routed dyadic pieces.

LoRA-Mixer adopts a hard-soft regime. During expert acquisition or joint expert training, a sample with domain label $d$ routes all tokens exclusively to expert $d$; during router optimization, all experts are softly fused with softmax scores; at inference, the system uses sparse Top-3 routing (Li et al., 17 Jun 2025). The same paper introduces Route-Specialization Balance Loss:

$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$0

which augments a global utilization term with token-level entropy structure so that routing is neither collapsed nor uniformly indiscriminate. The relevance to DoRA lies in the router regularization, not in the LoRA-specific update form.

LD-MoLE replaces fixed TopK routing with a differentiable Sparsegen projection. For token representation $$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$1, it computes expert logits $$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$2, predicts a token-specific sparsity factor $$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$3, and produces routing weights

$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$4

The support size $$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$5 is therefore adaptive, token-dependent, and layer-wise rather than fixed by a global hyperparameter (Zhuang et al., 30 Sep 2025). This is one of the strongest routing templates for a hypothetical DoRA-expert mixture because the routing mechanism is largely independent of the internal expert parameterization.

DMoERM supplies a different hybrid: outer sparse routing by task category, then inner dense fusion of all capability experts for the chosen task. The inner synthesis does not average scalar expert scores; it concatenates low-dimensional expert representations,

$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$6

and maps them to a final reward with a two-layer MLP (Quan, 2024). This demonstrates that expert composition can occur in representation space rather than only through scalar routing weights.

FourierMoE returns to sparse token-level routing, but the router is “frequency-adaptive” in the sense that selecting an expert selects a structured frequency band of adaptation. The paper uses token-level Top-$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$7 routing over frequency-specialized experts, with the final additive update given by

$$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$8

for the active expert set $$[\alpha_{\text{bb}}, \alpha_{\text{e}^1}, \ldots, \alpha_{\text{e}^N}] = g^i(x), \qquad g^i(x)=\operatorname{softmax}(\operatorname{Linear}(x)).$$9 (Jiang et al., 2 Apr 2026). The general lesson is that routing and expert structure should be co-designed.

4. Training regimes and optimization recipes

The cited literature supports several training recipes that could be transferred to a DoRA-expert mixture with minimal change to the routing logic. Modular Domain Experts uses a two-stage modular pipeline. In stage 1, a single expert is trained independently for each domain while the backbone remains frozen and only the expert layers and gating layers are updated. In stage 2, independently trained experts are composed into a single multi-domain model and then lightly fine-tuned on mixed-domain data. Freezing experts during composition is reported as more sample-efficient, whereas unfreezing them yields the strongest multi-domain performance (Schafhalter et al., 2024). This staged recipe is especially important because it decouples expert acquisition from expert composition.

Mixture-of-Distilled-Expert(s) follows the opposite strategy: a standard MoE forward pass is kept intact, and training is modified by adding mutual distillation,

$\mathbf{a}_j$0

with $\mathbf{a}_j$1 implemented as MSE between expert outputs, pairwise for two experts or against an average teacher for more than two (Xie et al., 2024). The method is not a PEFT recipe and has no relation to DoRA specifically, but it shows that expert interaction can be improved without changing inference-time architecture.

LoRA-Mixer emphasizes reusability. It supports direct deployment of pretrained, frozen LoRA modules from external repositories, including a Flan-T5 experiment in which five LoRAs from LoRAHub were mounted “without any modification” and only the router was trained on 2K mixed data (Li et al., 17 Jun 2025). “Flexible and Effective Mixing of LLMs into a Mixture of Domain Experts” pushes this further: Gate-less MOE assigns equal weight to each expert, and Noisy MOE uses a randomly initialized linear router with top-$\mathbf{a}_j$2 selection, both in a training-free setting (Lee et al., 2024). This is strong evidence that post hoc composition is a serious baseline, not merely a fallback.

DMoERM supplies a three-phase specialization recipe. First, a task-specific base model is fully fine-tuned. Second, one LoRA expert per capability point is trained on capability-specific preference labels generated by an external API. Third, the base and experts are frozen, and only an MLP that fuses expert representations is trained on overall pairwise preference labels (Quan, 2024). The architecture is reward-model specific, but the procedural logic—task specialization, expert specialization, frozen-expert synthesis—generalizes.

DR-LoRA introduces dynamic capacity growth rather than static expert sizing. It reserves parameter space up to a maximum LoRA rank $\mathbf{a}_j$3, activates only a small initial rank, and periodically expands selected experts based on a saliency score that combines routing frequency, rank importance, and a rank penalty:

$\mathbf{a}_j$4

Growth is quota-based, layer-local, and occurs only within a scheduled window after warmup (Deng et al., 8 Jan 2026). A plausible implication is that a DoRA-expert mixture could inherit the same allocation policy if “capacity” were redefined in DoRA terms.

5. Empirical performance in adjacent systems

The most direct modular-composition evidence comes from Modular Domain Experts. On Math + Code adaptation, the reported results are: full-parameter fine-tuning at Math 77.18, Code 67.89, English 47.95, Average 64.34; LoRA at Math 75.79, Code 65.71, English 49.05, Average 63.52; and MoDE 2×Experts at Math 77.47, Code 67.83, English 49.50, Average 64.93. Relative to full fine-tuning, the best MoDE variant is reported as +0.29 on Math, -0.06 on Code, +1.65 on English retention, and +0.59 on Average. The same paper reports that suitable MPMD sharding can reduce training step latency from 329 ms to 204 ms on 8 TPUv5e accelerators, which it describes as up to 38% faster than standard model parallelism in that setting (Schafhalter et al., 2024). These numbers do not validate DoRA experts, but they validate independently trained modular experts plus lightweight composition.

For multi-task PEFT, Mixture of Dyadic Experts reports 60.00 ROUGE-L with 6.64% added parameters for MoDE 16×4 on 756 English SNI tasks, versus 56.11 for LoRA 64 and 58.28 for MoLORA-SD 16×4; MoDE 6×4 reaches 60.91 ROUGE-L with only 2.69% additional parameters (Ning et al., 2024). LoRA-Mixer reports improvements over base models of 7.61% on GSM8K, 4.88% on HumanEval, and 3.08% on MedQA, and reports additional improvements of 1.09%, 1.45%, and 1.68% over state-of-the-art methods while using only 48% of the parameters in the compared setup (Li et al., 17 Jun 2025). These results show that adapter-based expert mixtures can remain parameter-efficient while outperforming single-adapter baselines.

For routing, LD-MoLE reports the highest average scores among compared LoRA-expert mixtures on both evaluated backbones. On Llama-3.2-3B, LD-MoLE reaches an average of 86.10, compared with 85.59 for ReMoLE and 83.27 for MoLA(2468). On Qwen3-1.7B, LD-MoLE reaches 85.10, versus 84.47 for ReMoLE and 82.94 for MoLA(2468). The same paper shows that learned $\mathbf{a}_j$5 outperforms fixed $\mathbf{a}_j$6 values: on Qwen3-1.7B, predicted $\mathbf{a}_j$7 yields 84.56 average, compared with 83.66 for $\mathbf{a}_j$8, 83.50 for $\mathbf{a}_j$9, and 83.35 for $\mathbf{b}_j^i$0 (Zhuang et al., 30 Sep 2025). This is direct evidence that routing quality can dominate the quality of the underlying expert family.

For hierarchical expert decomposition, DMoERM reports overall preference consistency of 70.7 for the full method at 1.8B, versus 67.0 for DMoERM-w/o-Outer and 62.6 for the best listed ensemble baselines. The paper also reports that, in roleplay, capability experts in Phase 2 reach roughly 80–86% test accuracy on their own capability labels, while the Phase 1 task-specific reward model is about 56% and the Phase 3 aggregated model reaches 68% (Quan, 2024). This indicates that lower-dimensional expert supervision can be easier to learn than overall preference.

Two adjacent papers are particularly informative about DoRA as a baseline. FourierMoE reports 88.19 average on Gemma 7B commonsense multitask with 0.03% trainable parameters, compared with 86.92 for DoRA at 0.14%; on LLaMA-3 8B math multitask, it reports 73.24 average at 0.01%, compared with 70.23 for DoRA around 0.11–0.12% (Jiang et al., 2 Apr 2026). DR-LoRA, operating on pretrained MoE LLMs rather than dense backbones, reports average benchmark scores of 42.6 versus 41.2 for DoRA on OLMoE and 66.4 versus 64.9 for DoRA on Phi-mini-MoE-instruct under matched average-rank budgets (Deng et al., 8 Jan 2026). These are not MoDE results, but they establish that expert structure, routing, and dynamic allocation can beat a static DoRA baseline.

6. Misconceptions, limits, and the open MoDE design space

The most common misconception is simply lexical: none of the cited “MoDE” papers is a “Mixture of DoRA Experts” paper. Modular Domain Experts uses transformer-layer experts, not DoRA-style decomposed updates (Schafhalter et al., 2024). Mixture of Dyadic Experts is explicitly a LoRA/MoE construction based on shared down-projection matrices and dyadic rank-one adapters, not DoRA (Ning et al., 2024). Mixture-of-Distilled-Expert(s) is a training regularizer for standard MoE, and Mixture of Deformation Experts is a dynamic 3D Gaussian Splatting method (Xie et al., 2024, Jin et al., 9 Jul 2026).

A second misconception is that any expert mixture that outperforms LoRA is therefore a better PEFT method in the same resource regime. Modular Domain Experts is the clearest counterexample: it beats LoRA and slightly exceeds full fine-tuning in average score and retention, but it does so with 2.376B total parameters versus 1.585B for LoRA and 1.583B for full fine-tuning (Schafhalter et al., 2024). Its claim is about modular expressivity and scaling behavior, not about matching LoRA’s parameter footprint.

A third misconception is that sophisticated learned routing is always necessary. The adapter-mixing toolkit shows that Gate-less equal-weight mixtures and Noisy top-$\mathbf{b}_j^i$1 mixtures are often strong baselines, and it repeatedly finds that router training is optional and most helpful on math-heavy tasks rather than uniformly beneficial (Lee et al., 2024). This is important because a future DoRA-expert mixture may inherit substantial complexity from the expert parameterization alone.

What remains open is the exact unit of composition for a DoRA mixture. The supplied literature repeatedly suggests that router logic is more transferable than expert algebra: LD-MoLE’s Sparsegen routing, DR-LoRA’s saliency-based capacity allocation, and LoRA-Mixer’s hard-soft schedule are all largely orthogonal to whether the expert is LoRA or DoRA (Zhuang et al., 30 Sep 2025, Deng et al., 8 Jan 2026, Li et al., 17 Jun 2025). A plausible implication is that a DoRA mixture could route complete DoRA experts, route only their directional component, or separately treat magnitude and direction, but none of the cited papers tests those choices directly.

Another open issue is expert non-redundancy. FourierMoE’s strongest architectural lesson is that experts become more useful when they inhabit explicitly different subspaces—here, distinct frequency bands with conjugate-symmetric complex coefficients (Jiang et al., 2 Apr 2026). This suggests that a DoRA-expert mixture may benefit from a similarly principled specialization axis rather than from a set of otherwise identical adapters. The cited literature does not yet specify what that axis should be for DoRA.

Taken together, the current record supports a narrow but clear conclusion. “Mixture of DoRA Experts” is best understood not as an established method name, but as an emerging design problem at the intersection of PEFT, modular composition, and expert routing. The literature already supplies viable blueprints for expert placement, routing, staged training, frozen-adapter reuse, hierarchical specialization, and dynamic capacity allocation. What it does not yet supply is a canonical paper in which those mechanisms are instantiated with DoRA experts themselves.