Accent-Aware Routing in ASR

• Accent-aware routing is a neural network strategy that integrates explicit accent information into mixture-of-expert architectures for improved specialization and ASR accuracy.
• It leverages accent-conditioned routers, expert-specific losses, and regularization techniques to ensure balanced and efficient expert utilization.
• Empirical results demonstrate significant reductions in error rates on both seen and unseen accents, highlighting its robustness and practical impact in speech recognition.

Accent-aware routing is a principled approach in neural network-based speech recognition that dynamically conditions routing decisions in mixture-of-experts (MoE) architectures on accent information, thereby enhancing the system's robustness and specialization across seen and unseen accents. A convergence of recent work demonstrates that explicitly incorporating accent labels, accent embeddings, or accent-informed selectors into neural routers yields consistent improvements, particularly for automatic speech recognition (ASR) exposed to a broad variety of speech accents.

1. Architectural Foundations and Routing Formulations

Accent-aware routing architectures typically build upon the Mixture-of-Experts paradigm, embedding explicit accent information within the routing network to modulate expert selection. Primary architectural components found across state-of-the-art systems include:

• Shared Encoder Backbone: A stack of acoustic encoder layers (FastConformer (Lee et al., 2 Feb 2026), Transformer (Prabhu et al., 2024), or LLM-based pipelines (Mu et al., 2024)) generates latent speech representations.
• MoE Modules: Inserted at selected layers, these comprise $N$ parallel expert sub-networks (feed-forward networks (FFNs), LoRA adapters, or codebook-conditioned projections).
• Accent-Conditioned Router: The router computes per-expert gating weights based on representations pooled from the encoder, concatenated with either one-hot accent labels, continuous accent embeddings, or output from a pre-trained accent recognizer.

In Moe-Ctc (Lee et al., 2 Feb 2026), for example, the router gating logits for each expert $j$ are:

$$L = W_g [x; a] + b_g, \quad \tilde L_{i,j} = L_{i,j} + \alpha \cdot 1[j=a_i]$$

with $$g_{i,j} = \frac{\exp(\tilde L_{i,j})}{\sum_{k=1}^N \exp(\tilde L_{i,k})}$$, where $x$ is the pooled hidden state, $a$ the one-hot accent vector, and $\alpha$ the accent prior strength.

SpeechMoE2 (You et al., 2021) generalizes this by constructing router inputs as the concatenation $[e^c; e^a; e^d; o^{l-1}]$, where $e^a$ is a global accent embedding, $e^d$ a domain embedding, and $o^{l-1}$ the previous layer's hidden activation, yielding expert gating via a standard softmax.

Accent-Specific Codebooks (Prabhu et al., 2024) replace a discrete MoE with cross-attention lookups into accent-indexed codebooks $$C^a = \{c^a_1, \ldots, c^a_M\} \in \mathbb{R}^{M \times d}$$, directly injecting accent-specific representation bias at multiple model layers.

HDMoLE (Mu et al., 2024) further extends this by hierarchical routing: expert mixture weights are computed via a combination of global, frozen accent recognizer scores and local, trainable routers with adaptive dynamic thresholding.

2. Expert Specialization, Training Objectives, and Regularization

Accent-aware routing strategies explicitly encourage or enforce the specialization of experts with respect to accent-specific acoustic or phonetic subspaces. This is achieved through:

• Expert-Level Losses: Moe-Ctc deploys expert-specific Connectionist Temporal Classification (CTC) heads, each incurring its own loss $$\mathcal{L}^{(\ell,j)}_{\mathrm{CTC}} = -\log p^{(\ell,j)}_{\mathrm{CTC}}(y^*|x)$$, encouraging each expert to minimize error on accent-routed data (Lee et al., 2 Feb 2026).
• Auxiliary Accent Recognition: Both one-hot and distributed accent embeddings are optimized through cross-entropy classification heads, e.g. $L_a = -\sum_u y^a_u \log \hat y^a_u$ in SpeechMoE2 (You et al., 2021).
• Routing Regularizers: Load-balancing and entropy penalties are imposed on the expert gating distribution to prevent expert collapse or under-utilization, e.g.,

$$\mathcal{L}_{\mathrm{aug}} = -\sum_j \bar{g}_j \log \bar{g}_j, \quad \bar{g}_j = \frac{1}{B} \sum_i g_{i,j}$$

as in (Lee et al., 2 Feb 2026), with analogous sparsity and balanced usage losses in (You et al., 2021, Mu et al., 2024).

• Dynamic Thresholds: HDMoLE introduces learnable thresholds $\tau_g, \tau_{\ell}$ for both global and local routers, adaptively activating a variable number of experts per input, further promoting cross-domain collaboration and robustness (Mu et al., 2024).

3. Training and Inference Procedures

Accent-aware routing systems implement a two-stage, curriculum-inspired schedule:

• Accent-Aware Training: During initial training, accent labels (or embeddings) are explicitly injected into the router—either as additive biases (as in Moe-Ctc), concatenated vectors (SpeechMoE2), or by selecting accent-indexed codebooks (Accent-Specific Codebooks). An auxiliary accent classification loss may be included.
• Label-Free Fine-Tuning and Inference: Once expert specialization stabilizes, accent label dependency in the routing is reduced or removed (by setting $\alpha=0$ and dropping accent-classification losses), allowing for generalization to unseen accents or label-free inference.
• Mixed and Hierarchical Routing: HDMoLE combines a frozen, global accent recognizer (routing at the utterance or batch level) with trainable, local routers within each layer. Adaptive thresholding modulates expert selection at both levels.

The final model routes incoming utterances, with or without accent labels, through sparse mixtures of experts, leveraging learned accent-condition specialization and generalization for improved transcription on both seen and novel accents.

4. Empirical Results and Performance Analysis

Accent-aware routing methods exhibit strong empirical results across a range of ASR benchmarks.

• Moe-Ctc (Lee et al., 2 Feb 2026):
  • Relative word error rate (WER) reductions of up to 29.3% on seen accents and 17.9% on unseen accents compared to strong FastConformer baselines.
  • Ablation studies reveal the critical importance of both accent bias injection and expert-level CTC heads.
• SpeechMoE2 (You et al., 2021):
  • Relative character error rate (CER) improvements of 1.9% to 17.7% across major Mandarin accent subsets with 2 or more experts per layer, compared to vanilla MoE.
  • Relative CER reductions scale up to 7.7% as the number of experts increases to 16, without increasing runtime.
• Accent-Specific Codebooks (Prabhu et al., 2024):
  • On Mozilla Common Voice, a 9% relative WER reduction over the best accent adaptation baseline for seen/unseen accents (overall WER: 21.19%).
  • Substantial gains hold in zero-shot generalization to out-of-domain non-native English (L2-Arctic).
• HDMoLE (Mu et al., 2024):
  • Achieves a KeSpeech CER of 16.58% with only 9.6% of the parameters of full projector fine-tuning, with minimal general-domain degradation.
  • Ablations confirm both hierarchical (global+local) routing and learnable thresholding are synergistic in driving performance.

A consistent observation is that accent-aware mechanisms support not just expert specialization, but also improved model invariance to phonetic surface-shifts induced by accent variations, as well as greater generalizability to accents not seen during training.

5. Variants, Extensions, and Practical Considerations

Accent-aware routing frameworks are highly modular and extensible:

• Router Input Variants: Accent information can be supplied by one-hot vectors, continuous embeddings learned via auxiliary classifiers, or output from dedicated accent recognizers. SpeechMoE2 further improves adaptability by integrating both domain and accent embeddings.
• Expert Definition: Experts can be dense FFNs, low-rank adaptation (LoRA) modules (HDMoLE), or discrete codebooks (Accent-Specific Codebooks). Soft gating, hard Top-K, or Top-1 routing are all compatible.
• Sparsity and Balance: Most architectures employ regularization to maintain balanced expert utilization and computational efficiency, e.g., sparsity losses, mean-importance penalties, or dynamic thresholds.
• Parameter and Computation Overhead: The increased number of experts or codebook parameters can be offset by sparse dispatch and highly efficient runtime kernels (e.g., FastMoE). For instance, SpeechMoE2 supports scaling to 16 experts per layer at no additional FLOPs (You et al., 2021), and accent-codebook methods add less than 0.1% parameter overhead to large Transformers (Prabhu et al., 2024). HDMoLE achieves competitive performance with <10% of full trainable parameter count (Mu et al., 2024).
• Generalization Strategies: Some systems interpolate between accent embeddings or utilize accent encoders to support recognition on continuous or unseen accent spectra (You et al., 2021).

6. Theoretical Insights and Mechanistic Interpretations

Empirical and analytical findings converge on several mechanisms by which accent-aware routing improves ASR robustness:

• Specialization emerges when experts receive gradient signals predominantly from utterances or frames associated with a specific accent, either due to explicit bias injection, cross-attention with accent-indexed codebooks, or accent-classification cross-entropy.
• Routing regularization (entropy, load-balancing) helps avoid collapse of expert assignment or over-reliance on a subset of experts, yielding a mixture of specialists and generalists.
• Intermediate losses (e.g., CTC supervision on each expert) ensure that accent-specific routing is tightly coupled to transcription accuracy, not just accent-class discriminability.
• Label-free inference is feasible post-training due to generalization properties encoded during earlier accent-aware or multitask optimization phases.
• Deployments benefit from runtime efficiency, as only a subset of experts is active for any utterance, and per-expert computational cost does not scale with the number of possible accents.

7. Summary of Impact and Empirical Gains

Accent-aware routing systematically enables neural ASR architectures to exhibit up to 30% relative error reduction on heavily accented speech compared to accent-agnostic or shallow ensemble baselines, with robust gains extending to unseen accent regimes. Core design choices—explicit router accent conditioning, expert-level auxiliary losses, and routing regularization—have proved to be key levers across model families, enabling both highly specialized and broadly generalizable ASR systems (Lee et al., 2 Feb 2026, Prabhu et al., 2024, Mu et al., 2024, You et al., 2021).