HDMoLE: Hierarchical Dynamic Mixture of LoRA Experts

Updated 6 July 2026

HDMoLE is a parameter-efficient adaptation method that employs multiple LoRA experts with hierarchical routing to address multi-accent ASR challenges while mitigating catastrophic forgetting.
It integrates a pretrained global accent recognizer with trainable local routers and dynamic thresholds to enable both domain specialization and cross-domain collaboration.
Empirical results demonstrate that HDMoLE achieves lower character error rates than single LoRA and other MoE methods, using only about 9.6% of the parameters required for full fine-tuning.

Searching arXiv for the primary HDMoLE paper and later work that reuses the method. HDMoLE, short for Hierarchical Dynamic Mixture of LoRA Experts, is a parameter-efficient fine-tuning method for LLM-based automatic speech recognition that combines low-rank adaptation with a mixture-of-experts routing scheme, adding hierarchical routing and dynamic thresholds to improve multi-domain adaptation while mitigating catastrophic forgetting (Mu et al., 2024). It was introduced for adapting a pre-trained Mandarin LLM-ASR system to multi-accent speech, where full supervised fine-tuning of all parameters is expensive and tends to degrade general-domain performance. In the original formulation, HDMoLE is applied to the projector module of a HuBERT-plus-Baichuan2 ASR stack, but its formulation is stated to generalize to any linear layer (Mu et al., 2024).

1. Problem setting and conceptual definition

HDMoLE was proposed against a specific failure mode of LLM-based ASR adaptation. Pre-trained LLM-ASR models trained on general speech corpora can perform poorly on multi-accent speech and other challenging domains, yet full projector fine-tuning or broader supervised fine-tuning imposes high computational cost and tends to induce catastrophic forgetting of the source or general domain (Mu et al., 2024). Standard parameter-efficient fine-tuning methods such as a single LoRA avoid modifying the base weights, but a single low-rank adapter is limited when one adapter must accommodate many accents or domains. Existing MoE-style LoRA methods, including MOELoRA, LoRAMoE, MoRAL, and MoA, treat each LoRA as an expert and use routing, but the paper identifies three recurrent issues: expert imbalance, unclear domain–expert correspondence, and the rigidity of a static Top- $K$ routing policy (Mu et al., 2024).

The core idea of HDMoLE is to augment each linear layer with multiple LoRA experts per layer and then control those experts through a two-level routing scheme. At the global level, a pretrained accent recognition model maps an utterance to an accent distribution, encouraging experts to specialize in accent domains. At the local level, each MoE layer has a trainable router that reads hidden features and refines expert weighting in a layer-specific manner. Dynamic thresholds replace fixed Top- $K$ selection, allowing each layer and input to activate a variable number of experts (Mu et al., 2024).

This design is explicitly aimed at two simultaneous objectives. The first is domain-specific specialization, in which accent-conditioned global routing provides a clearer correspondence between experts and accent domains. The second is cross-domain collaboration, in which local routers allow experts to share knowledge across accents when useful. The paper’s own summary characterizes the result as a domain-aware, dynamically sparse mixture-of-LoRA-experts framework (Mu et al., 2024).

2. Base architecture and LoRA-expert parameterization

In the original work, the underlying ASR system is a three-part LLM-ASR stack consisting of a HuBERT frontend, a projector module, and a Baichuan2 decoder LLM (Mu et al., 2024). The projector is a 4-layer Transformer with 256-dim MHSA, 2560-dim FFN, and 4 heads, and HDMoLE is inserted into the projector’s linear layers, specifically the FFN projections and the MHSA matrices $W_q, W_k, W_v, W_o$ (Mu et al., 2024). The formulation is stated to generalize to any linear weight $W$ .

The LoRA component follows the standard low-rank update parameterization. For an original weight matrix $W_0$ , the update is factorized as

$\Delta W = \frac{\alpha}{r}BA,$

where $A \in \mathbb{R}^{r \times d_{\text{in}}}$ , $B \in \mathbb{R}^{d_{\text{out}} \times r}$ , and $r \ll \min(d_{\text{in}}, d_{\text{out}})$ (Mu et al., 2024). The forward pass becomes

$\ell = W_0 x + \frac{\alpha}{r}BAx + b,$

with $K$ 0 and $K$ 1 frozen, $K$ 2 randomly initialized, and $K$ 3 initialized to zero (Mu et al., 2024).

HDMoLE extends this into a Mixture of LoRA Experts. Each expert $K$ 4 has its own $K$ 5, while all experts share the same frozen base weight $K$ 6. For input hidden features $K$ 7, the MoLE layer output is

$K$ 8

where $K$ 9 is the final routing weight for expert $W_q, W_k, W_v, W_o$ 0 after hierarchical routing and dynamic thresholding (Mu et al., 2024). In the original experiments, the number of LoRA experts per linear layer is $W_q, W_k, W_v, W_o$ 1, and the main experiments use LoRA rank $W_q, W_k, W_v, W_o$ 2, although ablations vary rank from 4 to 64 (Mu et al., 2024).

The practical significance of this parameterization is that the base ASR model remains frozen while adaptation is confined to a structured set of low-rank updates. A plausible implication is that this constrained adaptation space is part of why the method preserves general-domain performance better than full fine-tuning, although the paper attributes that effect directly to frozen base weights and structured low-rank adaptation rather than to an explicit theoretical bound (Mu et al., 2024).

3. Hierarchical routing and dynamic thresholding

The distinctive feature of HDMoLE is its hierarchical routing, which combines a global router and local routers (Mu et al., 2024). The global router is a pretrained accent recognition Conformer with 12 layers, trained on KeSpeech, achieving 81.44% accuracy on the KeSpeech test set, and remaining frozen during HDMoLE training and inference (Mu et al., 2024). For utterance-level speech features $W_q, W_k, W_v, W_o$ 3, it produces

$W_q, W_k, W_v, W_o$ 4

a probability distribution over experts (Mu et al., 2024). Because the AR model predicts accent-class probabilities, the global router is used to establish an explicit correspondence between experts and accent domains.

Each MoLE layer also contains a local router, implemented as a small trainable linear layer. For layer input $W_q, W_k, W_v, W_o$ 5, it produces

$W_q, W_k, W_v, W_o$ 6

again over the same set of experts (Mu et al., 2024). Unlike the global router, the local router is learned jointly with the LoRA experts on the ASR task. The intended division of labor is explicit in the paper: global routing drives domain specialization, while local routing captures fine-grained, layer-specific routing patterns and supports cross-domain collaboration (Mu et al., 2024).

HDMoLE then replaces fixed Top- $W_q, W_k, W_v, W_o$ 7 expert selection with dynamic thresholds. Each MoLE layer has a trainable global threshold $W_q, W_k, W_v, W_o$ 8 and local threshold $W_q, W_k, W_v, W_o$ 9, both initialized to $W$ 0 (Mu et al., 2024). The thresholded and renormalized global and local weights are

$W$ 1

$W$ 2

and the final adapted weight is

$W$ 3

If an expert falls below both thresholds, then $W$ 4 and the expert is inactive for that layer and input (Mu et al., 2024).

The paper emphasizes three consequences of this mechanism. First, the number of active experts is no longer governed by a single global hyperparameter $W$ 5. Second, each layer can learn a different sparsity regime. Third, the formulation keeps thresholds in the computational graph so they remain learnable by backpropagation (Mu et al., 2024). In later visual analysis, the original paper reports that lower projector layers activate more experts and higher layers activate fewer experts, which the authors interpret as early layers handling more generic and diverse acoustic features and later layers focusing on more specialized accent-specific patterns (Mu et al., 2024).

4. Training protocol and empirical results in multi-accent Mandarin ASR

The original HDMoLE model is trained only on KeSpeech, a ~1542 hour multi-accent Mandarin corpus with 27,237 speakers from 34 cities, including Standard Mandarin and eight major accented Mandarin varieties (Mu et al., 2024). The pre-trained HuBERT-plus-Baichuan2 ASR model had been trained on more than 11,000 hours of general-domain Mandarin data from WenetSpeech, AISHELL-1, AISHELL-2, and AISHELL-4, with no multi-accent Mandarin used for pretraining (Mu et al., 2024). AISHELL-2 is used only to measure retention of general-domain performance after accent adaptation (Mu et al., 2024).

During HDMoLE training, the trainable parameters are the LoRA expert matrices $W$ 6, the local router parameters for each MoLE layer, and the dynamic thresholds $W$ 7. The encoder, the projector’s base weights $W$ 8, the Baichuan2 LLM, and the AR global router remain frozen (Mu et al., 2024). The paper states that accent labels are needed only for training the AR model; HDMoLE itself is trained on standard ASR transcription labels, and no auxiliary expert-balancing loss is mentioned (Mu et al., 2024).

The main reported metric is character error rate. The pretrained HuBERT-plus-Baichuan2 model, without adaptation, yields 25.65 CER on KeSpeech and 3.50 CER on AISHELL-2 (Mu et al., 2024). Full projector fine-tuning trains 51M projector parameters and reaches 15.64 CER on KeSpeech but degrades to 4.91 CER on AISHELL-2 (Mu et al., 2024). A single LoRA uses 0.31M trainable projector parameters and reaches 19.98 CER on KeSpeech and 4.05 CER on AISHELL-2 (Mu et al., 2024). By contrast, HDMoLE uses 4.92M trainable projector parameters and reaches 16.58 CER on KeSpeech and 3.69 CER on AISHELL-2 (Mu et al., 2024). This is the basis for the claim that HDMoLE achieves performance close to full projector fine-tuning in the target multi-accent domain while using only ~9.6% of the trainable parameters required for that baseline and incurring only small degradation in the source domain.

Relative to MoE-LoRA baselines, the original paper reports the following KeSpeech and AISHELL-2 CER values: MOELoRA at 19.95 / 4.66, LoRAMoE at 18.76 / 4.33, MoRAL at 19.28 / 4.35, and MoA at 18.80 / 4.47, all worse than HDMoLE on both domains (Mu et al., 2024). The ablations are similarly direct. Removing dynamic thresholds degrades HDMoLE from 16.58 to 17.39 on KeSpeech and from 3.69 to 3.82 on AISHELL-2. Removing local routing degrades it to 18.33 / 3.86. Removing global routing degrades it to 18.76 / 3.94 (Mu et al., 2024). These values are used by the authors to argue that both levels of routing matter and that dynamic thresholding outperforms static Top- $W$ 9.

Two additional analyses sharpen the same conclusion. First, the quality of the global accent router matters strongly. When AR accuracy is 62.97%, the resulting KeSpeech CER is 18.11; at 81.44% AR accuracy it is 16.58; and with 100.0% oracle accent labels it becomes 15.35, slightly better than the 15.64 of full projector fine-tuning (Mu et al., 2024). Second, increasing LoRA rank improves performance but with saturation. The reported KeSpeech CER is 16.94 at rank 4, 16.58 at rank 8, 16.07 at rank 16, 15.92 at rank 32, and 15.98 at rank 64, with parameter counts ranging from 2.73M to 35.55M (Mu et al., 2024). The paper identifies rank 8 or 16 as a good efficiency-performance trade-off.

A later paper reuses HDMoLE in a different but related setting: multi-modal generative error correction for accented speech recognition (Mu et al., 12 Jul 2025). In that system, the base LLM is LLaMA-3.2-3B, the speech frontend is a Whisper-large-v3 encoder, and the model consumes both speech embeddings and a 1-best word-level hypothesis from Whisper (Mu et al., 12 Jul 2025). The authors first train separate mono-accent LoRA experts for Australian, Canadian, Chinese, German, Indian, Japanese, British, American, and South African accents, producing $W_0$ 0 experts (Mu et al., 12 Jul 2025). HDMoLE is then used to merge these frozen mono-accent experts into a single multi-accent GER model.

The architectural pattern remains recognizably the same. A pretrained accent recognition model supplies global weights, each layer has a trainable local router, and dynamic thresholds decide which accent experts are active (Mu et al., 12 Jul 2025). The later paper makes one additional distinction that was not foregrounded in the Mandarin ASR work: local routing can be implemented at either the utterance level or frame level, and the experiments report that frame-level local weights yield better WER (Mu et al., 12 Jul 2025). In this reuse of HDMoLE, the mono-accent LoRA parameters are frozen and only local routers and thresholds are trained (Mu et al., 12 Jul 2025).

The reported average WER values show the same pattern of advantage. On a combined multi-accent English dataset from Common Voice 19.0, VCTK, and AESRC, Whisper-large-v3 yields 6.34 AVG WER, fully fine-tuned Whisper yields 3.29, vanilla GER yields 3.26, multi-modal GER yields 3.20, mono-accent multi-modal GER yields 2.92, and HDMoLE yields 2.49 (Mu et al., 12 Jul 2025). When the HDMoLE-based model is followed by the paper’s multi-granularity GER, the average WER becomes 2.07, corresponding to a 67.35% relative WER reduction compared to the Whisper-large-v3 baseline (Mu et al., 12 Jul 2025). In a direct comparison against other MoLE methods using the same nine experts, the reported test average WERs are 3.55 for MOELoRA, 3.47 for MoRAL, 3.40 for MoLE/Wu, 3.22 for LoRAMoE, 3.33 for MoA, 2.57 for HDMoLE (utterance-level), and 2.49 for HDMoLE (frame-level) (Mu et al., 12 Jul 2025).

This later use is significant because it demonstrates that HDMoLE is not limited to projector-only adaptation in a HuBERT-plus-Baichuan2 ASR stack. The method is also used to combine accent-specific LoRA experts inside a generative correction pipeline, where it functions as a learned merger of frozen expert adapters rather than as a projector-only adaptation mechanism (Mu et al., 12 Jul 2025).

6. Scope, limitations, and nomenclature

The original paper identifies three best-use scenarios for HDMoLE: multi-accent or multi-dialect ASR in which a base model has been trained mainly on standard speech, multi-domain ASR where a global router can be constructed from a domain classifier, and low-resource adaptation budgets where full fine-tuning of large LLM-ASR models is impractical (Mu et al., 2024). The method’s efficiency comes from training only LoRA experts, local routers, and thresholds while leaving the base model untouched (Mu et al., 2024).

Its main limitations are equally explicit. HDMoLE depends on a global router whose quality strongly affects performance and which in turn requires labeled data for an accent or domain classifier (Mu et al., 2024). The later GER paper sharpens that limitation by reporting that when the accent recognizer is poor, with AR ACC ~50%, global routing can hurt performance, and that it becomes beneficial only after moderate accuracy, around 70%+ (Mu et al., 12 Jul 2025). The method also adds routing overhead, introduces additional hyperparameters such as rank, number of experts, and routing design choices, and does not use explicit stabilization mechanisms such as load balancing loss or noisy top- $W_0$ 1, although the papers argue that hierarchical routing and thresholding already improve expert utilization (Mu et al., 2024).

Future directions named in the original paper include extending HDMoLE beyond the projector to all components of LLM-based ASR (Mu et al., 2024). The later paper suggests applying the methods to larger foundation models and to more complex acoustic conditions such as noise and reverberation (Mu et al., 12 Jul 2025). This suggests a broader research program in which HDMoLE serves as a reusable expert-combination layer for multi-accent and multi-domain speech systems.

The term should also be distinguished from similarly named but unrelated work. “HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing” concerns deployment and parallel mapping for MoE inference on 3D near-memory processing accelerators, not LoRA-based ASR adaptation (Huang et al., 11 Sep 2025). “MoleHD: Ultra-Low-Cost Drug Discovery using Hyperdimensional Computing” is an HDC-based method for molecular property prediction and is likewise unrelated to hierarchical dynamic mixtures of LoRA experts for speech or LLMs (Ma et al., 2021). Within the speech literature, however, HDMoLE consistently denotes the Hierarchical Dynamic Mixture of LoRA Experts mechanism introduced for multi-accent LLM-ASR adaptation and later reused for accented generative error correction (Mu et al., 2024).