---
title: 'HDMoLE: Hierarchical Dynamic Mixture of LoRA Experts'
url: https://www.emergentmind.com/topics/hdmole
type: topic
---

# HDMoLE: Hierarchical Dynamic Mixture of LoRA Experts

Searching arXiv for the primary HDMoLE paper and later work that reuses the method.
HDMoLE, short for **Hierarchical Dynamic Mixture of LoRA Experts**, is a parameter-efficient fine-tuning method for LLM-based automatic speech recognition that combines low-rank adaptation with a mixture-of-experts routing scheme, adding **hierarchical routing** and **dynamic thresholds** to improve multi-domain adaptation while mitigating catastrophic forgetting [2409.19878]. It was introduced for adapting a pre-trained Mandarin LLM-ASR system to **multi-accent speech**, where full supervised fine-tuning of all parameters is expensive and tends to degrade general-domain performance. In the original formulation, HDMoLE is applied to the **projector module** of a HuBERT-plus-Baichuan2 ASR stack, but its formulation is stated to generalize to **any linear layer** [2409.19878].

## 1. Problem setting and conceptual definition

HDMoLE was proposed against a specific failure mode of LLM-based ASR adaptation. Pre-trained LLM-ASR models trained on general speech corpora can perform poorly on **multi-accent speech** and other challenging domains, yet **full projector fine-tuning** or broader supervised fine-tuning imposes high computational cost and tends to induce **catastrophic forgetting** of the source or general domain [2409.19878]. Standard parameter-efficient fine-tuning methods such as a **single LoRA** avoid modifying the base weights, but a single low-rank adapter is limited when one adapter must accommodate many accents or domains. Existing MoE-style LoRA methods, including **MOELoRA**, **LoRAMoE**, **MoRAL**, and **MoA**, treat each LoRA as an expert and use routing, but the paper identifies three recurrent issues: **expert imbalance**, **unclear domain–expert correspondence**, and the rigidity of a **static Top-\(K\)** routing policy [2409.19878].

The core idea of HDMoLE is to augment each linear layer with **multiple LoRA experts per layer** and then control those experts through a two-level routing scheme. At the **global level**, a pretrained **accent recognition** model maps an utterance to an accent distribution, encouraging experts to specialize in accent domains. At the **local level**, each MoE layer has a trainable router that reads hidden features and refines expert weighting in a layer-specific manner. Dynamic thresholds replace fixed Top-\(K\) selection, allowing each layer and input to activate a **variable number of experts** [2409.19878].

This design is explicitly aimed at two simultaneous objectives. The first is **domain-specific specialization**, in which accent-conditioned global routing provides a clearer correspondence between experts and accent domains. The second is **cross-domain collaboration**, in which local routers allow experts to share knowledge across accents when useful. The paper’s own summary characterizes the result as a **domain-aware, dynamically sparse mixture-of-LoRA-experts framework** [2409.19878].

## 2. Base architecture and LoRA-expert parameterization

In the original work, the underlying ASR system is a three-part LLM-ASR stack consisting of a **HuBERT frontend**, a **projector module**, and a **Baichuan2** decoder LLM [2409.19878]. The projector is a **4-layer Transformer** with **256-dim MHSA**, **2560-dim FFN**, and **4 heads**, and HDMoLE is inserted into the projector’s linear layers, specifically the FFN projections and the MHSA matrices \(W_q, W_k, W_v, W_o\) [2409.19878]. The formulation is stated to generalize to any linear weight \(W\).

The LoRA component follows the standard low-rank update parameterization. For an original weight matrix \(W_0\), the update is factorized as
\[
\Delta W = \frac{\alpha}{r}BA,
\]
where \(A \in \mathbb{R}^{r \times d_{\text{in}}}\), \(B \in \mathbb{R}^{d_{\text{out}} \times r}\), and \(r \ll \min(d_{\text{in}}, d_{\text{out}})\) [2409.19878]. The forward pass becomes
\[
\ell = W_0 x + \frac{\alpha}{r}BAx + b,
\]
with \(W_0\) and \(b\) frozen, \(A\) randomly initialized, and \(B\) initialized to zero [2409.19878].

HDMoLE extends this into a **Mixture of LoRA Experts**. Each expert \(i\) has its own \((A_i, B_i)\), while all experts share the same frozen base weight \(W_0\). For input hidden features \(H_{\text{in}}\), the MoLE layer output is
\[
H_{\text{out}} = W_0 H_{\text{in}} + \frac{\alpha}{r}\sum_{i=1}^{N} P_a^{(i)} B_i A_i H_{\text{in}} + b,
\]
where \(P_a^{(i)}\) is the final routing weight for expert \(i\) after hierarchical routing and dynamic thresholding [2409.19878]. In the original experiments, the number of LoRA experts per linear layer is **\(N=8\)**, and the main experiments use **LoRA rank \(8\)**, although ablations vary rank from **4 to 64** [2409.19878].

The practical significance of this parameterization is that the base ASR model remains frozen while adaptation is confined to a structured set of low-rank updates. A plausible implication is that this constrained adaptation space is part of why the method preserves general-domain performance better than full fine-tuning, although the paper attributes that effect directly to frozen base weights and structured low-rank adaptation rather than to an explicit theoretical bound [2409.19878].

## 3. Hierarchical routing and dynamic thresholding

The distinctive feature of HDMoLE is its **hierarchical routing**, which combines a **global router** and **local routers** [2409.19878]. The global router is a pretrained **accent recognition Conformer** with **12 layers**, trained on **KeSpeech**, achieving **81.44% accuracy on the KeSpeech test set**, and remaining **frozen during HDMoLE training and inference** [2409.19878]. For utterance-level speech features \(X_{\text{in}}\), it produces
\[
P_g = \mathrm{Softmax}(\mathrm{Router}_{\mathrm{Global}}(X_{\text{in}})),
\]
a probability distribution over experts [2409.19878]. Because the AR model predicts accent-class probabilities, the global router is used to establish an explicit correspondence between experts and accent domains.

Each MoLE layer also contains a **local router**, implemented as a small trainable linear layer. For layer input \(H_{\text{in}}\), it produces
\[
P_l = \mathrm{Softmax}(\mathrm{Router}_{\mathrm{Local}}(H_{\text{in}})),
\]
again over the same set of experts [2409.19878]. Unlike the global router, the local router is learned jointly with the LoRA experts on the ASR task. The intended division of labor is explicit in the paper: global routing drives **domain specialization**, while local routing captures **fine-grained, layer-specific routing patterns** and supports **cross-domain collaboration** [2409.19878].

HDMoLE then replaces fixed Top-\(K\) expert selection with **dynamic thresholds**. Each MoLE layer has a trainable **global threshold** \(\tau_g\) and **local threshold** \(\tau_l\), both initialized to \(1/N\) [2409.19878]. The thresholded and renormalized global and local weights are
\[
P_{ga}^{(i)} =
\frac{\mathbb{E}(P_g^{(i)} \ge \tau_g)\cdot P_g^{(i)}}{\sum_j \mathbb{E}(P_g^{(j)} \ge \tau_g)\cdot P_g^{(j)}} \cdot \tau_g,
\]
\[
P_{la}^{(i)} =
\frac{\mathbb{E}(P_l^{(i)} \ge \tau_l)\cdot P_l^{(i)}}{\sum_j \mathbb{E}(P_l^{(j)} \ge \tau_l)\cdot P_l^{(j)}} \cdot \tau_l,
\]
and the final adapted weight is
\[
P_a^{(i)} = P_{ga}^{(i)} + P_{la}^{(i)}.
\]
If an expert falls below both thresholds, then \(P_a^{(i)}=0\) and the expert is inactive for that layer and input [2409.19878].

The paper emphasizes three consequences of this mechanism. First, the number of active experts is no longer governed by a single global hyperparameter \(K\). Second, each layer can learn a different sparsity regime. Third, the formulation keeps thresholds in the computational graph so they remain learnable by backpropagation [2409.19878]. In later visual analysis, the original paper reports that **lower projector layers activate more experts** and **higher layers activate fewer experts**, which the authors interpret as early layers handling more generic and diverse acoustic features and later layers focusing on more specialized accent-specific patterns [2409.19878].

## 4. Training protocol and empirical results in multi-accent Mandarin ASR

The original HDMoLE model is trained **only on KeSpeech**, a **~1542 hour** multi-accent Mandarin corpus with **27,237 speakers from 34 cities**, including **Standard Mandarin** and **eight major accented Mandarin varieties** [2409.19878]. The pre-trained HuBERT-plus-Baichuan2 ASR model had been trained on **more than 11,000 hours** of general-domain Mandarin data from **WenetSpeech**, **AISHELL-1**, **AISHELL-2**, and **AISHELL-4**, with **no multi-accent Mandarin used for pretraining** [2409.19878]. **AISHELL-2** is used only to measure retention of **general-domain performance** after accent adaptation [2409.19878].

During HDMoLE training, the trainable parameters are the LoRA expert matrices \(\{A_i,B_i\}\), the local router parameters for each MoLE layer, and the dynamic thresholds \(\tau_g,\tau_l\). The encoder, the projector’s base weights \(W_0\), the Baichuan2 LLM, and the AR global router remain frozen [2409.19878]. The paper states that **accent labels are needed only for training the AR model**; HDMoLE itself is trained on standard ASR transcription labels, and **no auxiliary expert-balancing loss is mentioned** [2409.19878].

The main reported metric is **character error rate**. The **pretrained** HuBERT-plus-Baichuan2 model, without adaptation, yields **25.65 CER** on **KeSpeech** and **3.50 CER** on **AISHELL-2** [2409.19878]. **Full projector fine-tuning** trains **51M projector parameters** and reaches **15.64 CER** on KeSpeech but degrades to **4.91 CER** on AISHELL-2 [2409.19878]. A **single LoRA** uses **0.31M** trainable projector parameters and reaches **19.98 CER** on KeSpeech and **4.05 CER** on AISHELL-2 [2409.19878]. By contrast, **HDMoLE** uses **4.92M** trainable projector parameters and reaches **16.58 CER** on KeSpeech and **3.69 CER** on AISHELL-2 [2409.19878]. This is the basis for the claim that HDMoLE achieves performance close to full projector fine-tuning in the target multi-accent domain while using only **~9.6%** of the trainable parameters required for that baseline and incurring only **small degradation** in the source domain.

Relative to MoE-LoRA baselines, the original paper reports the following KeSpeech and AISHELL-2 CER values: **MOELoRA** at **19.95 / 4.66**, **LoRAMoE** at **18.76 / 4.33**, **MoRAL** at **19.28 / 4.35**, and **MoA** at **18.80 / 4.47**, all worse than HDMoLE on both domains [2409.19878]. The ablations are similarly direct. Removing **dynamic thresholds** degrades HDMoLE from **16.58** to **17.39** on KeSpeech and from **3.69** to **3.82** on AISHELL-2. Removing **local routing** degrades it to **18.33 / 3.86**. Removing **global routing** degrades it to **18.76 / 3.94** [2409.19878]. These values are used by the authors to argue that both levels of routing matter and that dynamic thresholding outperforms static Top-\(K\).

Two additional analyses sharpen the same conclusion. First, the quality of the global accent router matters strongly. When AR accuracy is **62.97%**, the resulting KeSpeech CER is **18.11**; at **81.44%** AR accuracy it is **16.58**; and with **100.0% oracle accent labels** it becomes **15.35**, slightly better than the **15.64** of full projector fine-tuning [2409.19878]. Second, increasing LoRA rank improves performance but with saturation. The reported KeSpeech CER is **16.94** at rank **4**, **16.58** at rank **8**, **16.07** at rank **16**, **15.92** at rank **32**, and **15.98** at rank **64**, with parameter counts ranging from **2.73M** to **35.55M** [2409.19878]. The paper identifies rank **8 or 16** as a good efficiency-performance trade-off.

## 5. Extension to multi-modal generative error correction for accented ASR

A later paper reuses HDMoLE in a different but related setting: **multi-modal generative error correction** for accented speech recognition [2507.09116]. In that system, the base LLM is **LLaMA-3.2-3B**, the speech frontend is a **Whisper-large-v3 encoder**, and the model consumes both **speech embeddings** and a **1-best word-level hypothesis** from Whisper [2507.09116]. The authors first train separate **mono-accent LoRA experts** for **Australian, Canadian, Chinese, German, Indian, Japanese, British, American, and South African** accents, producing **\(N=9\)** experts [2507.09116]. HDMoLE is then used to merge these frozen mono-accent experts into a single **multi-accent GER model**.

The architectural pattern remains recognizably the same. A pretrained **accent recognition** model supplies global weights, each layer has a trainable local router, and **dynamic thresholds** decide which accent experts are active [2507.09116]. The later paper makes one additional distinction that was not foregrounded in the Mandarin ASR work: local routing can be implemented at either the **utterance level** or **frame level**, and the experiments report that **frame-level local weights yield better WER** [2507.09116]. In this reuse of HDMoLE, the mono-accent LoRA parameters are frozen and only **local routers** and **thresholds** are trained [2507.09116].

The reported average WER values show the same pattern of advantage. On a combined multi-accent English dataset from **Common Voice 19.0**, **VCTK**, and **AESRC**, **Whisper-large-v3** yields **6.34 AVG WER**, **fully fine-tuned Whisper** yields **3.29**, **vanilla GER** yields **3.26**, **multi-modal GER** yields **3.20**, **mono-accent multi-modal GER** yields **2.92**, and **HDMoLE** yields **2.49** [2507.09116]. When the HDMoLE-based model is followed by the paper’s **multi-granularity GER**, the average WER becomes **2.07**, corresponding to a **67.35% relative WER reduction** compared to the Whisper-large-v3 baseline [2507.09116]. In a direct comparison against other MoLE methods using the same nine experts, the reported test average WERs are **3.55** for **MOELoRA**, **3.47** for **MoRAL**, **3.40** for **MoLE/Wu**, **3.22** for **LoRAMoE**, **3.33** for **MoA**, **2.57** for **HDMoLE (utterance-level)**, and **2.49** for **HDMoLE (frame-level)** [2507.09116].

This later use is significant because it demonstrates that HDMoLE is not limited to projector-only adaptation in a HuBERT-plus-Baichuan2 ASR stack. The method is also used to combine **accent-specific LoRA experts inside a generative correction pipeline**, where it functions as a learned merger of frozen expert adapters rather than as a projector-only adaptation mechanism [2507.09116].

## 6. Scope, limitations, and nomenclature

The original paper identifies three best-use scenarios for HDMoLE: **multi-accent or multi-dialect ASR** in which a base model has been trained mainly on standard speech, **multi-domain ASR** where a global router can be constructed from a domain classifier, and **low-resource adaptation budgets** where full fine-tuning of large LLM-ASR models is impractical [2409.19878]. The method’s efficiency comes from training only LoRA experts, local routers, and thresholds while leaving the base model untouched [2409.19878].

Its main limitations are equally explicit. HDMoLE depends on a **global router** whose quality strongly affects performance and which in turn requires labeled data for an accent or domain classifier [2409.19878]. The later GER paper sharpens that limitation by reporting that when the accent recognizer is poor, with **AR ACC ~50%**, global routing can **hurt** performance, and that it becomes beneficial only after moderate accuracy, around **70%+** [2507.09116]. The method also adds **routing overhead**, introduces additional hyperparameters such as **rank**, **number of experts**, and routing design choices, and does not use explicit stabilization mechanisms such as **load balancing loss** or **noisy top-\(K\)**, although the papers argue that hierarchical routing and thresholding already improve expert utilization [2409.19878].

Future directions named in the original paper include extending HDMoLE beyond the projector to **all components of LLM-based ASR** [2409.19878]. The later paper suggests applying the methods to **larger foundation models** and to more complex acoustic conditions such as **noise** and **reverberation** [2507.09116]. This suggests a broader research program in which HDMoLE serves as a reusable expert-combination layer for multi-accent and multi-domain speech systems.

The term should also be distinguished from similarly named but unrelated work. **“HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing”** concerns deployment and parallel mapping for MoE inference on **3D near-memory processing accelerators**, not LoRA-based ASR adaptation [2509.09420]. **“MoleHD: Ultra-Low-Cost Drug Discovery using Hyperdimensional Computing”** is an HDC-based method for molecular property prediction and is likewise unrelated to hierarchical dynamic mixtures of LoRA experts for speech or language models [2106.02894]. Within the speech literature, however, **HDMoLE** consistently denotes the **Hierarchical Dynamic Mixture of LoRA Experts** mechanism introduced for multi-accent LLM-ASR adaptation and later reused for accented generative error correction [2409.19878].

Source: https://www.emergentmind.com/topics/hdmole