Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM Framework: A Case Study of Multi-Domain Child-Adult ASR

Abstract: While Speech LLMs (Speech-LLMs) have achieved strong performance on adult Automatic Speech Recognition (ASR), their effectiveness on child speech remains under-explored, and single models often struggle to handle diverse adult and child age groups simultaneously. This paper proposes a Mixture-of-Experts (MoE) Speech-LLM for unified ASR across adult and child speech spanning diverse environments and age groups. The framework employs a Classifier-based Domain Router (C-DR) with a coarse-to-fine strategy and integrates both a Mixture-of-Projectors (MoP) and a Mixture-of-LoRAs (MoL) to model domain-specific variations. To address routing uncertainty near domain boundaries, an Entropy-Aware Routing (EAR) mechanism is introduced to dynamically incorporate a shared expert. Experiments on public child corpora demonstrate consistent improvements over baselines while preserving adult ASR performance. To our knowledge, this is the first work leveraging Speech-LLMs for unified, multi-domain ASR encompassing both children and adults.

• The paper introduces an entropy-aware domain-routed MoE framework that leverages hierarchical classification and dual-level domain specialization for unified child-adult ASR.
• It employs a classifier-based domain router alongside Mixture-of-Projectors and Mixture-of-LoRAs to mitigate domain confusion and achieve significant WER reductions.
• Empirical results on MyST, OGI-S, and LibriSpeech demonstrate improved performance over baselines, highlighting the framework's potential for scalable, multi-domain ASR.

Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM: Technical Overview and Analysis

Introduction

This paper addresses the persistent challenge of unified automatic speech recognition (ASR) across both adult and child speech domains, with a particular emphasis on the variability within child speech due to age and environment. Conventional Speech-LLMs, while effective for adult ASR, consistently underperform on child speech and struggle with domain balancing. The proposed framework introduces domain-specialized mixture-of-experts (MoE) within a Speech-LLM, augmented by classifier-based domain routing (C-DR) and a novel entropy-aware routing (EAR) scheme, explicitly constructed to mitigate domain confusion and boundary ambiguities. The methodology is validated using public child (MyST, OGI-S) and adult (LibriSpeech) corpora, demonstrating significant gains over strong baselines and prior approaches, especially in multi-domain and child-centric evaluations.

Methodology

The architecture integrates three core innovations:

1. Classifier-based Domain Router (C-DR) via Coarse-to-Fine Strategy: C-DR explicitly routes utterances to domain experts using supervised, coarse-to-fine hierarchical classification on encoder representations. This enables interpretable and controllable expert assignment, in contrast to conventional, unsupervised gating.
2. Hybrid Mixture-of-Experts Structure: Domain specialization occurs at both the acoustic frontend (Mixture-of-Projectors, MoP) and the LLM adaptation layer (Mixture-of-LoRAs, MoL), allowing both acoustic and linguistic domain variabilities to be captured. Each fine-grained or coarse domain is mapped to a dedicated projector–LoRA expert pair.
3. Entropy-Aware Routing (EAR): During inference, the routing entropy over expert assignments—reflecting classifier uncertainty—is computed and normalized. If uncertainty is high (i.e., an utterance lies close to domain boundaries or exhibits ambiguous features), EAR routes the input partially to a domain-agnostic shared expert. The final expert output is a weighted interpolation of the selected domain specialist and shared expert, parameterized by entropy.

   Figure 1: High-level workflow of the MoE Speech-LLM framework with C-DR routing and entropy-based shared expert integration.

Training is staged: C-DR and each expert are trained under ground-truth routing and domain supervision, with the backbone encoder and LLM frozen. The shared expert is trained separately using data pooled across domains, ensuring it represents average characteristics suitable for ambiguous cases.

Domain Classification Performance

The efficacy of the domain router substantially determines ultimate ASR performance, particularly under domain imbalance and acoustic overlap. The hierarchical, weighted-layer C-DR significantly outperforms top-layer and single-stage routers, especially for fine-grained (i.e., age-specific within child speech) distinctions. The confusion matrices reveal strong cross-dataset (adult vs. child, MyST vs. OGI-S) separability, with the main residual errors occurring for subtly distinct child age groups.

Figure 2: Comparative confusion matrices illustrating improved fine-grained classification with weighted-layer and coarse-to-fine C-DR strategies.

Notably, intermediate encoder layers encode substantial domain information, and learnable weighted-sum pooling yields more robust, generalizable feature representations for C-DR input.

Entropy-Aware Routing Dynamics

Routing entropy quantifies C-DR’s uncertainty for each test utterance. As shown in the entropy distributions, certainty is highest for older children, where acoustic features are most divergent, and lowest for younger age groups with overlapping developmental characteristics. EAR dynamically activates the shared expert proportional to this uncertainty.

Figure 3: Normalized routing entropy distributions stratified by child age group, highlighting pronounced uncertainty for younger ages.

This mechanism demonstrably improves ASR robustness for ambiguous utterances—particularly benefiting younger child speech, where age group labels are weak proxies for true acoustic phenotype due to developmental heterogeneity.

ASR Evaluation and Ablation

On OGI-S (age-segmented), MyST, and LibriSpeech test sets, the proposed C-DR MoE with EAR achieves the lowest word error rates (WER) among all evaluated methods that do not rely on upper-bound, domain-specific fine-tuning. Notably, joint MoP+MoL experts with hierarchical routing and EAR consistently outperform vanilla MoE, single-expert, and vanilla gating-based approaches across all domains without sacrificing adult ASR performance.

Key numerical results include:

• Substantial WER reductions for all OGI-S age groups with EAR, most notably for 4–7 year-olds (from 20.35% with single-expert to 17.64% with C-DR MoE+EAR under soft routing, a statistically significant gain).
• SOTA or near-SOTA results on MyST and LibriSpeech among unified, multi-domain models.
• Ablation reveals MoP alone is more beneficial than MoL alone, but both are needed for optimal performance; their combination outperforms either variant by ~1–2 WER absolute on child test sets.

Implications and Future Directions

The integration of classifier-supervised routing and entropy-aware expert selection establishes a new methodology for robust ASR under high domain variability and limited child speech data regimes. The route interpretability makes the system modular and amenable to domain generalization or incremental expert addition. The empirical observation that classifier-driven routing sometimes surpasses ground-truth (demographic) routing in ASR, especially among younger children, suggests that data-driven acoustic phenotypes can supersede nominal class definitions. This invites further exploration of unsupervised or latent domain discovery in future routing paradigms.

Practically, this framework can be extended to multi-language, accented, or noisy speech recognition tasks, wherever domain boundaries are not crisply defined. Theoretically, EAR embodies a class of uncertainty-aware expert combinations that could be studied alongside broader Bayesian routing or task-adaptive mixture models. Algorithmic advances may include tighter C-DR–LLM integration, improved parameter sharing strategies, or adaptive expert proliferation conditioned on empirical domain entropy.

Conclusion

The proposed entropy-aware domain-routed MoE Speech-LLM offers a robust and interpretable unification of adult and child ASR in a single architecture. By leveraging explicit hierarchical routing, dual-level domain specialization, and dynamic expert blending via entropy, the framework achieves strong, balanced performance across heterogeneous speech conditions. The methodology, validated on challenging age-stratified public corpora, sets a new direction for modular, domain-adaptive ASR system design and highlights key areas for extension to broader multi-domain multi-modal speech intelligence systems.