Mixture-of-LoRA (MoL)

• Mixture-of-LoRA (MoL) is a family of parameter-efficient methods that combine multiple LoRA updates through conditional routing, weighting, merging, or allocation for task-, token-, domain-, or key-specific adaptation.
• MoL methods retain a shared frozen backbone while activating selected low-rank experts, helping reduce interference in applications such as multimodal instruction tuning, speech recognition, forecasting, code generation, and watermarking.
• Effective MoL design requires balancing expert granularity, routing sparsity, adapter storage, dispatch overhead, load balancing, and specialization, since greater adaptation capacity does not guarantee universal gains or interpretable experts.

Mixture-of-LoRA (MoL), also called Mixture-of-LoRA Experts (MoLE), denotes a family of parameter-efficient adaptation methods that combine multiple Low-Rank Adaptation (LoRA) modules through conditional selection, weighting, merging, or allocation. A frozen pretrained model supplies shared computation, while several low-rank updates provide task-, domain-, token-, horizon-, speaker-, or key-dependent modifications. The central objective is to increase adaptation capacity and mitigate interference among heterogeneous requirements without replicating complete expert networks or updating the full backbone. Depending on the method, routing may be token-wise, instance-wise, layer-wise, rank-wise, projection-level, label-based, or deterministic from an external key. Representative systems include LLaVA-MoLE, MixLoRA, MoLE, SAML, MAS-LoRA, SMoRA, DynMoLE, RAMoLE, LoRA-Mixer, CoMoL, and MoL for recursive transformers.

1. Conceptual foundations and terminology

For a frozen weight matrix $W_0$, standard LoRA introduces a low-rank residual

$\Delta W = BA,$

where $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$, $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$, and $r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$. The adapted transformation is

$h = W_0x+\alpha BAx,$

with $\alpha$ controlling the update scale. Standard LoRA uses one fixed update for every input processed by the adapted module. Its parameter efficiency derives from replacing a dense $d_{\mathrm{out}}\times d_{\mathrm{in}}$ update with $r(d_{\mathrm{in}}+d_{\mathrm{out}})$ trainable parameters.

MoL generalizes this construction by maintaining multiple LoRA updates,

$\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$

and producing an input-dependent effective update. A generic formulation is

$\Delta W = BA,$0

where $\Delta W = BA,$1 are routing weights. The base model remains shared and frozen in the principal PEFT formulations, while the LoRA factors and routing parameters are trainable.

The term expert has several meanings in the literature:

• Complete LoRA expert: an independently parameterized pair $\Delta W = BA,$2, as in LLaVA-MoLE, MixLoRA, MAS-LoRA, and many MoLE systems.
• Rank-one expert: one outer-product component $\Delta W = BA,$3 of a single LoRA factorization, as in SMoRA.
• Core expert: a compact $\Delta W = BA,$4 matrix used with shared high-dimensional LoRA subspaces, as in CoMoL.
• Pretrained-model expert: an existing dense MoE expert to which LoRA capacity is allocated, as in DR-LoRA.
• Marker expert: a LoRA adapter whose activation encodes a watermark key rather than a task capability, as in MOLM.

MoL should therefore not be equated with a single routing architecture. Some systems use sparse top-1 or top-$\Delta W = BA,$5 dispatch, others use dense soft mixtures, and some use no learned router at all.

2. Data conflict, conditional adaptation, and expert granularity

The principal motivation for MoL is interference among heterogeneous objectives. In conventional multi-task LoRA, gradients from distinct domains accumulate in the same matrices $\Delta W = BA,$6 and $\Delta W = BA,$7. If the required transformations are incompatible, a shared update must compromise among them. LLaVA-MoLE studied general vision-language instruction following, document understanding, and biomedicine, reporting that a plain-LoRA eHub score of $\Delta W = BA,$8 on general data decreased to $\Delta W = BA,$9 after document data were added and to $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$0 after biomedical data were added (Chen et al., 2024).

MoL introduces conditional capacity:

$A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$1

or, for a soft mixture,

$A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$2

Different inputs can consequently activate different adaptation directions. The frozen base transformation retains common behavior, while the conditional residuals provide specialization. Empirical results across multimodal instruction tuning, language understanding, code generation, speech recognition, forecasting, and watermarking support the narrower conclusion that conditional low-rank updates can reduce some forms of interference. They do not establish that each expert becomes a clean, globally interpretable domain module.

Expert granularity is a major design variable. Module-level MoLE partitions adaptation into comparatively large LoRA blocks. SMoRA instead observes that a multi-LoRA system can be rewritten as a single LoRA with block-diagonal rank gating. If expert $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$3 has rank $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$4 and total rank is $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$5, concatenated factors $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$6 and $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$7 yield

$A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$8

SMoRA refines this partition to individual rank-one components:

$A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$9

where $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$0 activates $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$1 of $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$2 ranks. Its principal configuration uses total rank $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$3 and activates $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$4 ranks per token. On FLAN-v2, SMoRA-64-8 achieved averages of $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$5 for Llama-2-7B and $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$6 for Llama-2-13B, exceeding dense LoRA-64 scores of $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$7 and $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$8, respectively (Zhao et al., 25 Jan 2025).

The granularity trade-off is not monotonic. Coarse modules provide efficient dispatch but restrict combinations; rank-wise routing enables finer selective sharing but introduces fragmented computation and router overhead. In MoL for recursive transformers, token-conditional FFN modulation restores functional diversity lost through recursive parameter sharing while retaining the shared backbone (Nouriborji et al., 14 Dec 2025). In time-series forecasting, segment-specific mixtures provide horizon-dependent adaptation without requiring an independent LoRA for every forecast step (Pan et al., 23 May 2025).

3. Routing mechanisms

Token-wise sparse routing

LLaVA-MoLE attaches multiple LoRA experts to MLP/FFN linear layers and uses a top-1 router:

$B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$9

The selected residual is added to the frozen FFN output:

$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$0

The router is shared across the FFN’s linear sublayers, while self-attention uses ordinary LoRA. Tokens are grouped by selected expert, processed using the corresponding low-rank matrices, and scattered back to their original positions. The method uses a load-balancing objective with coefficient $r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$1 and a capacity of 4096 tokens, so tokens are not dropped (Chen et al., 2024).

MixLoRA uses top-$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$2 routing over eight FFN LoRA experts, with the principal configuration selecting two experts per token. Selected probabilities are normalized and the corresponding expert outputs are combined. Independent ordinary LoRA modules are also applied to attention projections $r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$3 (Li et al., 2024).

DynMoLE replaces fixed top-$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$4 routing with entropy-dependent selection. It computes router probabilities, evaluates Tsallis entropy

$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$5

and applies soft routing to high-entropy tokens and Top-$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$6/Top-$r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$7 routing to low-entropy tokens. Its reported configuration uses $r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$8, entropy threshold $r\ll\min(d_{\mathrm{in}},d_{\mathrm{out}})$9, $h = W_0x+\alpha BAx,$0, and minimum $h = W_0x+\alpha BAx,$1. On Llama-2-7B commonsense benchmarks, DynMoLE reached $h = W_0x+\alpha BAx,$2 average accuracy versus $h = W_0x+\alpha BAx,$3 for MoLA and $h = W_0x+\alpha BAx,$4 for LoRA (Li et al., 1 Apr 2025).

LD-MoLE replaces nondifferentiable Top-$h = W_0x+\alpha BAx,$5 with Sparsegen. A learned token-dependent scalar $h = W_0x+\alpha BAx,$6 controls the projection

$h = W_0x+\alpha BAx,$7

subject to $h = W_0x+\alpha BAx,$8. The number of active experts is therefore determined by both the logits and $h = W_0x+\alpha BAx,$9, rather than fixed globally. Sparsegen guarantees at least one active expert and provides a closed-form, piecewise-differentiable routing rule (Zhuang et al., 30 Sep 2025).

Dense and layer- or instance-conditioned mixtures

MoLE composes independently trained LoRAs using hierarchical gates. Expert outputs are computed at network, block, layer, or matrix granularity and combined by temperature-controlled softmax weights:

$\alpha$0

The base block output remains an explicit branch. Unlike token-wise sparse MoE, the demonstrated mechanism is primarily input-conditioned and layer-specific rather than a separate gate for every token (Wu et al., 2024).

RAMoLE introduces a retrieve-then-compose architecture for dynamically changing adapter pools. LoraRetriever uses representative examples of each uploaded LoRA to construct an embedding and retrieves the top-$\alpha$1 candidates by cosine similarity. A RouterLoRA then computes soft attention weights over the retrieved candidates. The system therefore combines hard pool-level retrieval with soft candidate-level routing and is designed to accommodate newly uploaded LoRAs without retraining an expert-specific routing embedding (Zhao et al., 2024).

CoMoL performs routing in the LoRA latent space. It factorizes each expert as

$\alpha$2

where $\alpha$3 and $\alpha$4 are shared and $\alpha$5 is expert-specific. The token is projected to $\alpha$6, routed using an $\alpha$7 router, and the selected core matrices are merged before one shared high-dimensional transformation. The resulting parameter complexity is

$\alpha$8

rather than $\alpha$9 (Cao et al., 28 Feb 2026).

Label-based and deterministic routing

MAS-LoRA trains one expert per accent using hard accent labels. At inference, an unknown accent uses a uniform mixture, while a known accent receives increased but non-exclusive weight. No learned router improved the reported results. On L2-ARCTIC, MAS-LoRA with accent-specific encoder experts and ordinary decoder LoRA achieved WER $d_{\mathrm{out}}\times d_{\mathrm{in}}$0, compared with $d_{\mathrm{out}}\times d_{\mathrm{in}}$1 for ordinary LoRA and $d_{\mathrm{out}}\times d_{\mathrm{in}}$2 for full fine-tuning (Bagat et al., 26 May 2025).

MOLM uses deterministic binary-key routing for generative watermarking. An $d_{\mathrm{out}}\times d_{\mathrm{in}}$3-bit key is divided into chunks, each selecting one of $d_{\mathrm{out}}\times d_{\mathrm{in}}$4 adapters in one of $d_{\mathrm{out}}\times d_{\mathrm{in}}$5 routed blocks:

$d_{\mathrm{out}}\times d_{\mathrm{in}}$6

The route remains fixed throughout diffusion denoising. In the default Stable Diffusion configuration, $d_{\mathrm{out}}\times d_{\mathrm{in}}$7, $d_{\mathrm{out}}\times d_{\mathrm{in}}$8, and the key size is 28 bits. This is key-conditioned marker composition rather than data-dependent task routing (Fares et al., 30 Sep 2025).

4. Parameterization, computation, and systems design

MoL increases stored adapter parameters even when activated computation remains sparse. A standard LoRA module for one linear layer contains approximately

$d_{\mathrm{out}}\times d_{\mathrm{in}}$9

parameters. With $r(d_{\mathrm{in}}+d_{\mathrm{out}})$0 experts, this becomes approximately

$r(d_{\mathrm{in}}+d_{\mathrm{out}})$1

plus router parameters. Thus, claims that MoL has the same parameter count as LoRA generally concern activated computation or the frozen backbone, not total stored trainable adapter parameters.

Top-1 and top-$r(d_{\mathrm{in}}+d_{\mathrm{out}})$2 methods can keep expert-specific computation near ordinary LoRA because only selected low-rank paths are evaluated. LLaVA-MoLE reported broadly similar scores for dense and sparse two-expert variants, but dense two-expert routing used 83% of GPU memory versus 61% for sparse MoLE; dense three-expert routing ran out of memory on a long-context three-dataset mixture (Chen et al., 2024). MixLoRA similarly combines sparse FFN experts with independent attention LoRA and reports that m-LoRA reduces GPU memory consumption by 40% and token computation latency by 30% in its high-throughput implementation (Li et al., 2024).

Sparse execution is not automatically faster. Routing, token gathering, scattering, kernel launches, and load imbalance can offset theoretical FLOP reductions. SMoRA therefore uses an indexed sparse matrix multiplication kernel, indexed_matmul, implemented with TVM/CUDA. RAMoLE uses batch-level unions of retrieved adapters and a mapping matrix to execute heterogeneous requests jointly. CoMoL avoids repeated high-dimensional expert transformations by merging compact core matrices before projection. These systems illustrate that routing algorithms and execution kernels must be co-designed.

SAML uses a different efficiency strategy. It forms weighted aggregate LoRA matrices,

$r(d_{\mathrm{in}}+d_{\mathrm{out}})$3

and computes

$r(d_{\mathrm{in}}+d_{\mathrm{out}})$4

This avoids separate matrix multiplications for every expert, although it uses soft routing and does not establish the same sparse-computation behavior as top-$r(d_{\mathrm{in}}+d_{\mathrm{out}})$5 MoL (Zhao et al., 2024).

LoRA-Mixer routes at projection matrices rather than replacing complete FFN or attention blocks. It supports Transformer projections and Mamba/SSM projections, including input, output, $r(d_{\mathrm{in}}+d_{\mathrm{out}})$6, and $r(d_{\mathrm{in}}+d_{\mathrm{out}})$7 projections. Its training uses soft routing, while inference uses Top-3 routing. The method also supports frozen external LoRAs and introduces a Specialization Balance Loss that combines utilization consistency with entropy-based selectivity (Li et al., 17 Jun 2025).

5. Applications and empirical evidence

MoL has been evaluated in multiple settings with different definitions of expert specialization.

Multimodal instruction tuning: LLaVA-MoLE mitigates conflicts among general, document, and biomedical image-text instructions. On general-plus-document data, its two-expert configuration achieved $r(d_{\mathrm{in}}+d_{\mathrm{out}})$8 eHub, $r(d_{\mathrm{in}}+d_{\mathrm{out}})$9 ChartQA, and $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$0 DocVQA, compared with $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$1 eHub, $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$2 ChartQA, and $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$3 DocVQA for plain LoRA. MixLoRA reported consistent improvements over conventional LoRA across MME and additional unseen multimodal datasets, including an approximately $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$4 MME improvement for $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$5 relative to LoRA with rank 32 (Chen et al., 2024, Shen et al., 2024).

Multi-task language modeling and reasoning: MixLoRA with eight experts and top-2 routing achieved average accuracies of $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$6 on Llama-2-7B, $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$7 on Llama-2-13B, and $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$8 on Mistral-7B in the reported multi-task benchmarks. DynMoLE, LD-MoLE, CoMoL, and LoRA-Mixer respectively focus on entropy-adaptive routing, differentiable variable-support routing, core-space merging, and projection-level routing. CoMoL achieved $\Delta W_i=B_iA_i,\qquad i=1,\ldots,N,$9 average accuracy on Qwen3-8B mathematical reasoning with $\Delta W = BA,$00 million trainable parameters, compared with $\Delta W = BA,$01 for LoRA with $\Delta W = BA,$02 million parameters (Li et al., 2024, Zhuang et al., 30 Sep 2025, Cao et al., 28 Feb 2026).

Dynamic adapter libraries: RAMoLE addresses Uploadable Machine Learning, where the adapter pool changes after router training. On Llama-2-7B, it achieved aggregate scores of $\Delta W = BA,$03 in IID evaluation and $\Delta W = BA,$04 in OOD evaluation, compared with $\Delta W = BA,$05 and $\Delta W = BA,$06 for parameter fusion. Its distinguishing property is that new adapters can be indexed using representative examples rather than requiring fixed expert IDs (Zhao et al., 2024).

Speech recognition: SAML combines speaker-adaptive LoRA experts with NF4 quantization. After approximately sevenfold model-size reduction, SAML-NF4 achieved WER $\Delta W = BA,$07 for Whisper on LibriSpeech-SA versus $\Delta W = BA,$08 for the original FP32 model, and WER $\Delta W = BA,$09 for the Conformer system versus $\Delta W = BA,$10 for its original FP32 model. MAS-LoRA uses a different label-routed accent formulation and reports improved unknown-accent WER and reduced catastrophic forgetting (Zhao et al., 2024, Bagat et al., 26 May 2025).

Time-series forecasting: MoLA uses shared LoRA experts with segment-specific mixing vectors. One-step pretraining avoids the multi-output decoder’s expressiveness bottleneck, and horizon segments receive distinct mixtures. On nine forecasting datasets, MoLA with iTransformer achieved the best MSE in 33 of 45 reported cases and the best MAE in 36 of 45 cases (Pan et al., 23 May 2025).

Recursive transformers: MoL modifies shared FFNs in ModernALBERT with token-level top-2 mixtures of LoRA-modulated FFN outputs. ModernALBERT models between 50M and 120M parameters achieved GLUE averages from $\Delta W = BA,$11 to $\Delta W = BA,$12, and MoL exceeded MoA in a controlled comparison, obtaining $\Delta W = BA,$13 versus $\Delta W = BA,$14 (Nouriborji et al., 14 Dec 2025).

Generative watermarking: MOLM uses LoRA routes as distributed marker codes. On Stable Diffusion, its default 28-bit configuration reported clean bit accuracy $\Delta W = BA,$15, FID $\Delta W = BA,$16, and robustness to crop, rotation, resizing, brightness, and JPEG distortions. Its purpose is not task adaptation but key-dependent attribution and verification (Fares et al., 30 Sep 2025).

6. Limitations, unresolved issues, and scope

MoL methods share several unresolved technical issues.

Routing semantics are heterogeneous: token-wise top-$\Delta W = BA,$17 routing, sequence-level routing, instance retrieval, accent labels, horizon segments, and binary keys are not interchangeable. A result for one routing granularity does not establish equivalent behavior for another.

Stored and activated costs differ: multiple experts increase adapter storage, optimizer state, and often activation memory. Sparse routing can reduce expert-specific computation but may add dispatch overhead. Dense mixtures may avoid irregular kernels while evaluating all experts. Several papers provide qualitative or partial efficiency evidence rather than complete FLOP, latency, energy, and peak-memory accounting.

Load balancing can conflict with specialization: auxiliary losses in LLaVA-MoLE, MixLoRA, DynMoLE, and LD-MoLE encourage utilization across experts, while SMoRA uses bias updates and LoRA-Mixer uses specialization-aware balancing. Uniform utilization is not necessarily optimal when task frequencies are nonuniform. Conversely, SAML and MoL for recursive transformers do not report conventional load-balancing objectives, and SAML observes expert collapse in some layers.

Expert interpretability is limited: routing visualizations often show partial, layer-dependent, or task-correlated specialization rather than one expert per domain. LLaVA-MoLE explicitly reports that no expert is consistently preferred across every layer and domain. Routing clusters and activation patterns are empirical evidence of correlation, not proof of semantic disentanglement.

Expert count and rank require tuning: too few experts or ranks may underfit; too many can starve experts of training data, complicate optimization, or degrade performance. SMoRA found $\Delta W = BA,$18 favorable in its principal setting, while CoMoL performance was not monotonic with expert count. DynMoLE and LD-MoLE introduce additional sensitivity to entropy, sparsity, and support-size parameters.

Architectural compatibility constrains composition: RAMoLE and LoRA-Mixer require compatible base architectures, projection shapes, and insertion points. Adapters trained for different architectures, ranks, or PEFT mechanisms cannot generally be reused directly.

Dynamic routing complicates merging: a token-dependent mixture cannot usually be represented exactly by one globally merged LoRA. Some systems provide approximate or restricted merging: ModernALBERT MoL uses uniform or EMA-weighted merging, while MAS-LoRA can merge its uniform unknown-accent mixture because the coefficients are fixed. Merging generally sacrifices conditional specialization.

Evidence remains benchmark-dependent: reported gains establish improvements under the stated datasets, models, and training procedures, not universal superiority over full fine-tuning or arbitrary adapter combinations. Several studies lack full-finetuning baselines, complete ablations, detailed statistical uncertainty, or large-scale deployment measurements.

Overall, MoL is best understood as a design space rather than a single algorithm. Its common principle is to factor adaptation into reusable low-rank components and condition their use on input, task, layer, rank, horizon, speaker, or key. The main research trade-off is between shared capacity and specialization: ordinary LoRA shares one update indiscriminately, full MoE replicates large expert networks, and MoL seeks intermediate structures in which low-rank adaptation capacity is increased while the pretrained backbone remains shared.