LID-Posterior Driven LoRA Routing in mASR

• The paper introduces an end-to-end LID-posterior-driven routing mechanism that dynamically integrates language-specific LoRA experts with mHuBERT-CTC for single-pass, language-agnostic decoding.
• It details a hierarchical transformer structure where lower shared layers compute LID posteriors and upper layers apply soft-gated expert adaptation, optimizing both ASR and LID jointly.
• Empirical evaluations demonstrate that the method slightly improves word error rates and significantly boosts LID accuracy while halving the inference passes compared to two-stage systems.

LID-posterior-driven LoRA routing is an inference mechanism developed for hierarchical Low-Rank Adaptation Mixture-of-Experts (LoRA-MoE) architectures in multilingual automatic speech recognition (ASR). Operating within a Connectionist Temporal Classification (CTC) framework, this method enables language-agnostic, single-pass decoding by dynamically routing information through language-specific LoRA expert modules, gated by the model’s internally computed language identification (LID) posterior probabilities. This routing approach is central to the HLoRA framework, which is integrated into the mHuBERT-CTC backbone to achieve efficient and scalable domain adaptation for low-resource mASR while obviating the need for external language labels or multi-stage inference procedures (Zheng et al., 2 Jan 2026).

1. Computation of the LID Posterior in mHuBERT-CTC

The LID-posterior-driven routing begins by embedding a lightweight LID classifier at layer $k$ of the frozen mHuBERT-CTC encoder. Let $X_h\in\mathbb{R}^{T \times d}$ denote the frame-level features after propagation through the shared LoRA-adapted lower $k$ layers:

$$X_h = \mathcal{F}_1(\mathrm{CNN}(X), W_s, \theta_{\mathrm{trans}}^{1..k})$$

The output $X_h$ is projected via a linear transformation followed by softmax to yield per-frame language logits and posteriors:

$$Y_L = X_h \cdot W^\top_{\mathrm{LID}} + b_{\mathrm{LID}}, \quad Y_L \in \mathbb{R}^{T \times L}$$

$$p_{t,i} = \mathrm{softmax}_i(Y_L[t]) = \frac{\exp(Y_L[t,i])}{\sum_{j=1}^L \exp(Y_L[t,j])}$$

Final LID posterior vector $p \in \Delta^{L-1}$ is obtained by mean-pooling across time:

$p_i = \frac{1}{T} \sum_{t=1}^T p_{t,i}$

This posterior $p$ serves as the soft routing distribution, governing expert utilization in later transformer layers (Zheng et al., 2 Jan 2026).

2. Hierarchical LoRA-MoE Model Structure

The model partitions its transformer stack into $N$ total layers, divided into $k$ shared lower layers and $N-k$ upper expert layers.

• The shared multilingual LoRA ($W_s$) is injected into the $Q,K,V$ projections for the initial $k$ layers, with parameterization:

$$\Delta W_s = \frac{A_s B_s}{\alpha}, \quad A_s \in \mathbb{R}^{r \times d},\ B_s \in \mathbb{R}^{d \times r}$$

$r=32$, $\alpha=64$.

• Language-specific LoRA expert modules $\{ W_L^i \}_{i=1..L}$ are similarly structured and inserted in upper layers $(\ell > k)$, providing adaptation capacity tailored to each supported language:

$\Delta W_L^i = \frac{A_L^i B_L^i}{\alpha}$

• For each layer $\ell > k$, transformer weights are formed as:

$$W_{\mathrm{eff}} = W_{\mathrm{base}} + \sum_{i=1}^L g_i \Delta W_L^i + \Delta W_s$$

where $g_i$ is the LID posterior for language $i$, serving as expert gating.

This design enables the model to learn both language-invariant acoustic mappings and language-dependent feature refinements (Zheng et al., 2 Jan 2026).

3. LID-Posterior-Driven Routing: Soft-Gating and Inference Procedure

The routing mechanism employs a soft-gating mixture at each expert layer, weighted by the internal LID posterior. Specifically, at layer $\ell > k$:

$$\Delta W_{\mathrm{total}} = \Delta W_s + \sum_{i=1}^L p_i \Delta W_L^i$$

The transformer block then operates with parameters $W_{\mathrm{base}} + \Delta W_{\mathrm{total}}$ for all remaining forward passes.

Single-pass inference can be summarized as:

Input: speech X 1. Extract features through frozen CNN. 2. For ℓ=1…k: H_ℓ = TransformerLayer(H_{ℓ−1}; W_base, ΔW_s) 3. Aggregate H_k over time → Ḣ_k 4. Compute LID logits: Y_LID = Ḣ_k·W_LIDᵀ + b_LID 5. Compute posterior p = softmax(Y_LID) 6. Initialize H = H_k 7. For ℓ=k+1…N: ΔW_total = ΔW_s + Σ_{i=1}^L p[i]·ΔW_L^i H ← TransformerLayer(H; W_base, ΔW_total) 8. Compute CTC‐score = CTCHead(H; W_CTC_base + Σ_i p[i]·ΔW_CTC^i) 9. Decode best path with beam/greedy CTC Output: transcription

The system thus integrates LID and expert adaptation into a unified forward computation (Zheng et al., 2 Jan 2026).

4. Model Specification and Computational Properties

The standard configuration employs:

• $N=12$ transformer layers (mHuBERT-147), with $k=6$ shared.
• $L=5$ primary languages (EN-IN, FR, DE, JA, KO), with 6 additional “other” languages for LID.
• LoRA rank $r=32$, $\alpha=64$ for $Q,K,V$ projections and CTC layer.
• Hidden dimension $d \approx 1024$.
• Parameter overhead: $W_s$ adds $\sim$1.5M, each expert $\sim$0.2M, total $\sim$5M extra parameters, overall model size $\sim$102M vs 97M baseline.

Inference overhead is minimal: a single additional linear transform and softmax (for $\sim$11 languages), negligible compared to transformer computation. Critically, elimination of a second forward pass (required in two-stage systems) approximately halves encoder-side computational cost (Zheng et al., 2 Jan 2026).

5. Decoding Efficiency and Empirical Accuracy

Evaluation on the MLC-SLM test set demonstrates that LID-posterior-driven routing achieves word error rates (WER) and LID accuracies on par with or exceeding prior two-stage LoRA-based adaptation, while substantially improving decoding efficiency. The empirical results are summarized in the following table:

ID	System	LID Rep.	Passes	WER (%)
S4	mHuBERT-CTC + two-stage LoRA	✗	2	24.8
S6	mHuBERT-CTC-HLoRA (ours)	✗	1	24.7

• HLoRA attains WER 24.7%, slightly better than two-stage (24.8%), with half the inference passes.
• LID accuracy increases from 90.1% (two-stage) to 97.9% (HLoRA) as a consequence of end-to-end joint optimization.
• Ablation over $k$ supports $k=6$ as the optimal shared-expert split ($\sim$26.0% WER on dev/test) (Zheng et al., 2 Jan 2026).

6. Language-Agnostic Single-Pass Decoding

LID-posterior-driven LoRA routing leverages only the internally computed LID posterior, obviating the need for external language identity annotations or separate LID modules at inference. This approach eliminates error propagation between sequential LID/ASR phases, precludes manual language switching or metadata requirements, and preserves on-device, low-latency operation by requiring only a single forward pass. The resulting system embodies true language-agnostic multilingual ASR, jointly learning ASR and LID without auxiliary supervision or post-hoc rerouting (Zheng et al., 2 Jan 2026).