Mixture-of-LoRA-Experts (MoLE)

• Mixture-of-LoRA-Experts (MoLE) is a parameter-efficient framework that adapts frozen pretrained models with multiple low-rank updates combined via learnable gating.
• It spans applications in NLP, vision, speech recognition, and more by dynamically routing expert adaptations based on task, token, or instance characteristics.
• MoLE incorporates diverse routing and anti-collapse strategies to balance expert specialization while mitigating feature or task interference.

Mixture-of-LoRA-Experts (MoLE) denotes a family of parameter-efficient fine-tuning architectures in which a frozen pretrained model is adapted not by a single low-rank update, but by multiple low-rank adapters treated as experts and combined through a routing mechanism. In the literature, MoLE appears both as a method for fusing separately trained LoRAs and as a general recipe for task-conditional, token-conditional, instance-conditional, or architecture-conditional adaptation. The design space now spans NLP, vision and language, diffusion, speech recognition, transformer architecture search, few-shot classification, continual multimodal instruction tuning, and video summarization. The original formulation emphasized hierarchical control and unfettered branch selection, while later work diversified the routing rules, sparsity mechanisms, regularization strategies, and expert parameterizations (Wu et al., 2024, Chen et al., 2024, Zhu et al., 2024, Jeon et al., 8 May 2026).

1. Origins, motivation, and scope

The earliest explicit MoLE formulation in the supplied literature was introduced to address the limitations of direct arithmetic fusion and reference-tuning fusion for multiple LoRAs. In that setting, a naïve merged model is written as

$\hat W \;=\; W \;+\;\sum_{i=1}^N \Delta W_i,$

or, with normalized fusion,

$$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$

The criticism was twofold: direct merging can drift the pretrained model too far from its pretrained manifold, while normalized fusion shrinks each contribution and can wash out important features learned by individual LoRAs. The response was to treat each LoRA update at a block as an expert, and to learn a gate that mixes expert outputs at a finer granularity than whole-model arithmetic composition (Wu et al., 2024).

That initial motivation broadened rapidly. In multimodal instruction tuning, MoLE was proposed to mitigate data conflicts when mixing instruction data from distinct domains, using sparse top-1 routing over LoRA experts in the MLP layers (Chen et al., 2024). In human-centric diffusion, MoLE was used to mix face and hand LoRA experts trained on specialist close-up data (Zhu et al., 2024). In transformer architecture search, Mixture-of-LoRA-Experts was introduced as a way to enable subnet-specific feature learning inside a weight-sharing supernet and to mitigate feature collapse (Jeon et al., 8 May 2026). This suggests that MoLE is better understood as a design space for structured low-rank adaptation than as a single architecture.

2. Core mathematical structure

A common starting point is LoRA itself. Given a frozen linear layer with weight $W$, LoRA learns a low-rank update $\Delta W$ while keeping $W$ fixed. One representative formulation is

$$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$

with $A\in\mathbb R^{d_{\rm out}\times r}$, $B\in\mathbb R^{d_{\rm in}\times r}$, and $r\ll \min(d_{\rm in},d_{\rm out})$. In MoLE, the single low-rank update is replaced by multiple expert updates and a gate. A representative layer equation is

$$y^l \;=\; W^l x^l \;+\;\sum_{k=1}^K p^l_k\,\bigl(E_k^l\,x^l\bigr),$$

where $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$0 is the $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$1-th LoRA expert and $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$2 is the corresponding gating weight (Jeon et al., 8 May 2026).

Another common form, used in diffusion-oriented work, is

$$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$3

where $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$4. The basic abstraction is therefore stable across papers: a frozen backbone term plus a weighted sum of low-rank expert outputs (Xiao et al., 25 Dec 2025).

What varies is the expert parameterization and insertion locus. LLaVA-MoLE replaces each single LoRA adapter by $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$5 parallel LoRA experts in every linear sub-layer of the FFN, while keeping the attention layers unchanged (Chen et al., 2024). MixLoRA inserts LoRA-based experts within the feed-forward network block of a frozen pretrained dense model and also uses independent attention-layer LoRA adapters for the query, key, value and output projections (Li et al., 2024). DR-LoRA adapts each expert’s linear submodules in a pretrained MoE LLM but allows heterogeneous ranks $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$6 under a fixed total budget (Deng et al., 8 Jan 2026). CoMoL shares low-rank basis matrices across experts and stores each expert in a compact core matrix $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$7, producing updates of the form $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$8 (Cao et al., 28 Feb 2026). RotMoLE retains the standard MoE-LoRA form but adds a learned rotation $$\hat W \;=\; W \;+\;\sum_{i=1}^N w_i\,\Delta W_i, \quad \sum_{i=1}^N w_i=1.$$9 so that the update becomes $W$0 rather than only scalar reweighting (Sun et al., 25 May 2026).

The original MoLE paper also made the granularity of gating explicit. It described network-wise, block-wise, and matrix-wise control, with block-wise or layer-wise gating reported as the best empirical choice in that work (Wu et al., 2024).

3. Routing, specialization, and anti-collapse mechanisms

Routing is the most variable component in the MoLE literature. The original MoLE used a block-wise gating function that concatenates and normalizes expert outputs, projects them to scores, and applies a softmax with learnable temperature $W$1 (Wu et al., 2024). LLaVA-MoLE adopted a simple linear router $W$2, formed a softmax distribution, and executed only the top-1 expert for each token, with no hard capacity limits and with capacity set to the full context length so that no tokens are dropped (Chen et al., 2024). Human-centric diffusion used both local gating, $W$3, and global gating, $W$4, combining spatially localized and globally pooled signals (Zhu et al., 2024).

Later work diversified routing further. TAS-LoRA conditioned routing on the architecture of the sampled subnet rather than on its input features. Its router encodes each transformer block’s discrete attributes, processes them sequentially with a one-layer LSTM, and outputs a routing matrix $W$5 over the Q, K, V, and MLP sub-layers (Jeon et al., 8 May 2026). InstructMoLE replaced token-level routing with Instruction-Guided Routing derived from the user’s comprehensive instruction, selecting a single expert council per instance and broadcasting that choice across all tokens; the stated motivation was to avoid patch-wise routing jitter, spatial fragmentation, and semantic drift (Xiao et al., 25 Dec 2025). DynMoLE introduced a hybrid routing strategy that switches between dense and sparse behavior according to the Tsallis entropy of the router distribution, using soft routing for uncertain tokens and Top-$W$6/Top-$W$7 routing otherwise (Li et al., 1 Apr 2025). LD-MoLE replaced non-differentiable TopK with a differentiable routing function based on Sparsegen and a token-dependent sparsity controller $W$8, allowing the model to adaptively determine the number of experts to activate for each token at different layers (Zhuang et al., 30 Sep 2025). HDMoLE used hierarchical routing for multi-accent ASR, with a frozen global accent-recognition router and a learnable local router, followed by dynamic thresholds $W$9 and $\Delta W$0 rather than a static Top-$\Delta W$1 rule (Mu et al., 2024).

Because routers can collapse onto a few experts or keep experts functionally redundant, the literature also contains several anti-collapse devices. The original MoLE used a balance loss

$\Delta W$2

to prevent single-expert dominance (Wu et al., 2024). LLaVA-MoLE, MixLoRA, DynMoLE, InstructMoLE, and LD-MoLE all included load-balancing losses in different forms (Chen et al., 2024, Li et al., 2024, Li et al., 1 Apr 2025, Xiao et al., 25 Dec 2025, Zhuang et al., 30 Sep 2025). InstructMoLE added an output-space orthogonality loss

$\Delta W$3

to encourage the experts’ functional outputs to span a high-rank basis (Xiao et al., 25 Dec 2025). TAS-LoRA took a different route: it added no extra regularizer and instead relied on group-wise router initialization, setting $\Delta W$4 and biasing blocks in group $\Delta W$5 toward expert $\Delta W$6 with $\Delta W$7, thereby forcing each expert to be claimed by one group at the outset (Jeon et al., 8 May 2026). DR-LoRA used a saliency score

$\Delta W$8

to decide which experts should grow in rank under a fixed parameter budget (Deng et al., 8 Jan 2026).

A common misconception is that MoLE necessarily means token-level Top-$\Delta W$9 routing. The supplied literature directly contradicts that view: it includes block-wise soft gating, top-1 token routing, local-plus-global routing, architecture-conditioned routing, global instruction-guided routing, dynamic-threshold routing, and differentiable dynamic sparsity (Wu et al., 2024, Chen et al., 2024, Zhu et al., 2024, Jeon et al., 8 May 2026, Xiao et al., 25 Dec 2025, Mu et al., 2024, Zhuang et al., 30 Sep 2025).

4. Efficiency, merging, and parameter allocation

MoLE is consistently framed as a parameter-efficient alternative to full fine-tuning, but the way efficiency is realized differs by application. In TAS-LoRA, only the LoRA experts and router are trainable, and for a chosen subnet the expert mixture is merged into the frozen layer weight at inference: $W$0 which yields zero extra cost at deployment (Jeon et al., 8 May 2026). LLaVA-MoLE emphasized that sparse top-1 activation keeps the training and inference cost roughly constant compared to the original LoRA method; the router adds $W$1 extra FLOPs, and sparse MoLE at $W$2 uses about $W$3 of a single A100-80GB’s memory headroom versus $W$4 for a dense-Mixture-of-LoRA or $W$5 for plain LoRA (Chen et al., 2024). InstructMoLE reported peak GPU memory $W$6 versus $W$7 for token-level Expert Choice, with zero latency overhead at inference (Xiao et al., 25 Dec 2025).

Several papers focus on parameter growth directly. DR-LoRA fixes the total budget to match a uniform-rank LoRA of rank $W$8 while redistributing rank across experts, so that $W$9 matches the fixed budget exactly (Deng et al., 8 Jan 2026). MoLA varies the number of LoRA experts by layer rather than using the same count everywhere, with the central empirical claim that higher layers need more experts; the supplied summary states that the inverted-triangle allocation can use much fewer parameters than the setting with the same number of experts in every layer and that $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$0 total experts across 32 layers gives $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$1 fewer expert parameters than uniform 8-expert layers (Gao et al., 2024). CoMoL compresses experts by sharing bases and storing only core matrices, giving

$$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$2

which is described as making CoMoL’s footprint nearly constant in $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$3 because $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$4 (Cao et al., 28 Feb 2026).

In speech recognition, SAML combines MoLE with block-wise NF4 quantisation. The base model is compressed by $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$5, while a 10-expert SAML layer adds only $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$6 parameters rather than the tens of millions associated with a conventional FFN-MoE with dense experts. For Whisper, the full model is $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$7, a single LoRA is $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$8, and SAML with 10 experts and $$W' \;=\; W \;+\;\Delta W, \qquad \Delta W \;=\; A\,B^\top,$$9 is $A\in\mathbb R^{d_{\rm out}\times r}$0 trainable (Zhao et al., 2024).

The literature also shows that deployment behavior is not uniform. Some variants merge experts into static weights for zero-cost inference, some keep sparse routing at runtime, and some, such as D-MoLE, activate current-task experts together with selected prior-task experts under a parameter budget (Jeon et al., 8 May 2026, Ge et al., 13 Jun 2025). This suggests that “MoLE efficiency” refers not to one fixed systems profile but to a spectrum of PEFT trade-offs.

5. Empirical landscape

The empirical record in the supplied papers is broad rather than concentrated in a single benchmark family. The table lists representative results that are explicitly reported.

Setting	Representative result	Source
V&L LoRA fusion	Text-align $A\in\mathbb R^{d_{\rm out}\times r}$1, Image-align $A\in\mathbb R^{d_{\rm out}\times r}$2	(Wu et al., 2024)
NLP LoRA fusion	Translation $A\in\mathbb R^{d_{\rm out}\times r}$3 BLEU; BBH $A\in\mathbb R^{d_{\rm out}\times r}$4 EM	(Wu et al., 2024)
Human-centric diffusion	SD v1.5 HPS $A\in\mathbb R^{d_{\rm out}\times r}$5, IR $A\in\mathbb R^{d_{\rm out}\times r}$6	(Zhu et al., 2024)
MLLM instruction tuning	eHub $A\in\mathbb R^{d_{\rm out}\times r}$7, ChartQA $A\in\mathbb R^{d_{\rm out}\times r}$8 on full $A\in\mathbb R^{d_{\rm out}\times r}$9 mixture	(Chen et al., 2024)
Commonsense reasoning	DynMoLE full average $B\in\mathbb R^{d_{\rm in}\times r}$0 vs LoRA $B\in\mathbb R^{d_{\rm in}\times r}$1	(Li et al., 1 Apr 2025)
Transformer architecture search	AutoFormer-T $B\in\mathbb R^{d_{\rm in}\times r}$2	(Jeon et al., 8 May 2026)
Transfer from AutoFormer-S	CIFAR-100 $B\in\mathbb R^{d_{\rm in}\times r}$3	(Jeon et al., 8 May 2026)
Quantised ASR adaptation	Whisper-SAML-adaptation-NF4 Libri-SA WER $B\in\mathbb R^{d_{\rm in}\times r}$4, TL3-SA WER $B\in\mathbb R^{d_{\rm in}\times r}$5	(Zhao et al., 2024)
Multi-accent ASR	HDMoLE $B\in\mathbb R^{d_{\rm in}\times r}$6 on KeSpeech / AISHELL-2	(Mu et al., 2024)

Beyond these examples, the papers report strong task-specific gains in several other settings. In the original MoLE paper, V&L three-subject composition outperformed NLA and SVDiff, and NLP evaluations on FLAN-T5 showed MoLE ahead of LoRAHub and PEMs on translation, struct-to-text, closed-book QA, BBH, and NLI (Wu et al., 2024). Human-centric diffusion reported consistent gains in HPS and IR on COCO Human Prompts and DiffusionDB Human Prompts, plus a three-stage ablation showing that naïve plug-in of experts hurts until the gating layers are trained (Zhu et al., 2024). LLaVA-MoLE reported that on the mixed datasets, MoLE can even outperform the plain-LoRA baseline trained with twice the samples, and that increasing LoRA rank partly mitigates conflict but incurs $B\in\mathbb R^{d_{\rm in}\times r}$7 memory (Chen et al., 2024). TAS-LoRA reported improvements not only on ImageNet but also on Flowers, Cars, and INAT-19, with the stated interpretation that Mixture-of-LoRAExperts and group-wise router initialization mitigate feature collapse effectively (Jeon et al., 8 May 2026).

The range of applications is itself notable. MoLE variants appear in LLM fine-tuning, multimodal instruction tuning, human-centric image generation, multi-conditional diffusion, ViT supernets, ASR, video summarization, continual multimodal learning, few-shot concept learning, multilingual generation, and multilingual NLI (Li et al., 2024, Xiao et al., 25 Dec 2025, Du et al., 8 Mar 2025, Ji et al., 5 Jun 2025, Sun et al., 25 May 2026).

6. Misconceptions, limitations, and open directions

A recurrent misconception is that MoLE is merely arithmetic LoRA merging. The foundational MoLE paper was explicitly motivated by the failure modes of arithmetic fusion and by the cost and inflexibility of reference tuning, and it introduced learnable block-wise gating precisely to avoid those limitations (Wu et al., 2024). A second misconception is that more experts monotonically improve performance. The evidence is mixed: LLaVA-MoLE reported that $B\in\mathbb R^{d_{\rm in}\times r}$8 improves eHub from $B\in\mathbb R^{d_{\rm in}\times r}$9, but a further increase to $r\ll \min(d_{\rm in},d_{\rm out})$0 yields a slight drop because of insufficient tokens per expert (Chen et al., 2024). InstructMoLE stated that activating too many experts $r\ll \min(d_{\rm in},d_{\rm out})$1 increases interference and that too few $r\ll \min(d_{\rm in},d_{\rm out})$2 under-utilizes the expert pool (Xiao et al., 25 Dec 2025). The original MoLE summary also noted that with very large expert counts, for example $r\ll \min(d_{\rm in},d_{\rm out})$3, overall performance can still drop (Wu et al., 2024).

The central technical limitation across papers is some form of collapse or mismatch: feature collapse in supernets, task interference in diffusion transformers, router uncertainty in LLM fine-tuning, expert starvation, representational collapse, or domain conflict in mixed instruction data (Jeon et al., 8 May 2026, Xiao et al., 25 Dec 2025, Li et al., 1 Apr 2025, Chen et al., 2024). Different papers address different parts of that problem: group-wise initialization, orthogonality loss, Tsallis-entropy regularization, dynamic thresholds, saliency-based rank growth, and hierarchical routing all target distinct failure modes rather than a single universal pathology (Jeon et al., 8 May 2026, Xiao et al., 25 Dec 2025, Li et al., 1 Apr 2025, Mu et al., 2024, Deng et al., 8 Jan 2026).

Several open directions are stated directly in the supplied sources. DR-LoRA notes that validation is still needed for expert-choice, dynamic mixtures, and multimodal MoEs, and that scaling to $r\ll \min(d_{\rm in},d_{\rm out})$4 MoEs remains to be explored (Deng et al., 8 Jan 2026). RotMoLE highlights dynamic selection of multiple rotation planes, multi-angle rotations per expert, and combining rotational gating with other router regularizers (Sun et al., 25 May 2026). D-MoLE frames continual multimodal instruction tuning as architectural evolution under parameter budgets, with dynamic task adaptation and modality balancing as primary challenges (Ge et al., 13 Jun 2025). CoMoL argues for fine-grained token-level routing with parameter efficiency comparable to standard LoRA through dynamic core space merging (Cao et al., 28 Feb 2026). LD-MoLE emphasizes adaptive, token-dependent, and layer-wise expert allocation rather than a fixed $r\ll \min(d_{\rm in},d_{\rm out})$5 for all tokens and all layers (Zhuang et al., 30 Sep 2025).

Taken together, these results suggest a stable high-level thesis. MoLE is not one router, one sparsity pattern, or one parameterization; it is a structured PEFT framework in which the low-rank update itself becomes a routed object. The literature then asks three recurring questions: how experts are parameterized, how experts are selected, and how specialization is induced without losing the efficiency that motivates LoRA in the first place.