H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR

Abstract: Multi-talker Automatic Speech Recognition (MTASR) faces significant challenges in accurately transcribing overlapping speech, particularly under complex high-overlap conditions. While recent Mixture-of-Experts (MoE) approaches have shown promise, they typically rely on frame-independent routing that leads to temporal myopia, and depend solely on the downstream ASR objective, which results in implicit and ungrounded representation learning. To address these limitations, we propose Holistic Speaker-Aware Guided Experts (H-SAGE) for MoE-based MTASR. Specifically, we introduce a Speaker-Aware Global Encoder to capture long-term dependencies, supervised by an auxiliary Overlap-Aware Loss that explicitly guides the model to discern acoustic states. Furthermore, we design a Holistic Gating Mechanism to arbitrate expert selection by jointly evaluating global context and local details. Experiments on LibriSpeechMix demonstrate that H-SAGE achieves consistent improvements over strong baselines, particularly in complex scenarios, validating that explicit acoustic guidance effectively enhances expert collaboration. Our code can be found at https://github.com/NKU-HLT/H-SAGE.

• The paper's main contribution is the integration of explicit acoustic state modeling using a Speaker-Aware Global Encoder and Overlap-Aware Loss to improve expert routing and transcription accuracy in multi-talker scenarios.
• The novel holistic gating mechanism synthesizes local and global cues to dynamically balance expert activation, addressing temporal myopia in previous MoE models.
• Quantitative evaluations on LibriSpeechMix benchmarks demonstrate significant improvements in high-overlap conditions, validating the model's robust performance and zero-shot generalization.

H-SAGE: Holistic Speaker-Aware Guided Experts for Mixture-of-Experts Based Multi-Talker ASR

Introduction and Motivation

Multi-talker ASR remains a challenging paradigm due to the necessity of robustly disentangling and transcribing overlapping speech signals, especially under high-overlap conditions. Traditional ASR systems, while effective in single-speaker scenarios, exhibit significant performance degradation in the presence of speaker overlap. Existing Mixture-of-Experts (MoE) approaches, such as GLAD, have introduced limited frame-independent global routing but suffer from temporal myopia and rely on downstream ASR objectives, resulting in implicit and insufficiently discriminative acoustic modeling.

H-SAGE addresses these deficiencies by introducing explicit acoustic state modeling and robust expert selection strategies. The architecture augments MoE-based SISO MTASR systems with a Speaker-Aware Global Encoder (SA-Encoder) for holistic global context extraction and a Holistic Gating Mechanism that synthesizes local and global cues to arbitrate expert activation. These innovations are further supervised by a novel Overlap-Aware Loss, enforcing explicit recognition of acoustic states crucial for disentangling overlapping speech.

Figure 1: Overview of the H-SAGE architecture integrating the Conformer Encoder with MoLE blocks, a Speaker-Aware Global Encoder, and the Holistic Gating Mechanism, all jointly optimized with explicit overlap-aware supervision.

H-SAGE Architecture

Speaker-Aware Global Encoder and Explicit Overlap Modeling

The SA-Encoder operates directly on the feature representations obtained from the convolutional frontend, preserving critical fine-grained acoustic information. By employing a multi-head self-attention block followed by a feed-forward module, the SA-Encoder models long-range temporal dependencies required for reliable identification of complex speaker activity states, such as single-speaker and overlap regions.

To provide explicit semantic grounding, H-SAGE introduces an auxiliary Overlap-Aware Loss, $\mathcal{L}_{OA}$, which trains the SA-Encoder to produce frame-level overlap-aware state predictions (0: padding, 1: single-speaker, 2: overlap). These targets, automatically derived from mixture boundaries, ensure that the extracted global representations are directly aligned with acoustic conditions relevant to permutation-invariant decoding and expert routing.

Figure 2: The Overlap-Aware Loss mechanism enforces explicit frame-level supervision of speaker state, crucial for robust expert activation in overlapping regions.

Holistic Gating Mechanism

Expert routing within the MoE framework is realized via the Holistic Gating Mechanism, which overcomes information asymmetry by concatenating local frame-level cues with SA-Encoder–derived global context. The resulting holistic representation parameterizes soft fusion weights governing the balance between local and global expert activation probabilities. This dynamic arbitration ensures that expert selection is contextually sensitive to both instantaneous phonetic features and evolving long-range acoustic states, which is essential for robust transcription under highly entangled speaker scenarios.

Multi-Task Optimization and Loss Formulation

Training employs multi-task learning, combining the standard ASR loss ($\mathcal{L}_{ASR}$) with the Overlap-Aware Loss ($\mathcal{L}_{OA}$), modulated by hyperparameter $\lambda$. This weighting is empirically tuned to maintain an optimal balance between transcription accuracy and explicit speaker/activity disentanglement.

Experimental Evaluation

Dataset and Experimental Protocol

H-SAGE is evaluated on the LibriSpeechMix (LSM) benchmark, which provides both 2-speaker and 3-speaker (zero-shot) conditions. The dataset is systematically stratified into low, mid, and high-overlap levels, enabling granular performance analysis. All models (including SOT, SOT-SACTC, GLAD-SOT, and various ablation baselines) are matched for parameter count to ensure fair comparison.

Quantitative Results

H-SAGE establishes new state-of-the-art performance across high-overlap and generalization (LSM-3mix) tasks, with the following strong numerical results:

• On LSM-2mix (Test, OA-WER): H-SAGE achieves 6.2%, outperforming all baselines (e.g., GLAD-SOT at 6.8%).
• On LSM-3mix (Test, OA-WER): H-SAGE obtains 19.8%, maintaining superiority in robust generalization over unseen speaker counts.

Figure 3: H-SAGE consistently outperforms competitive baselines on LSM-2mix, especially as overlap severity increases.

Notably, H-SAGE's improvements are most significant in the high-overlap regime, validating the effectiveness of explicit overlap modeling and holistic expert arbitration. While SACTC excels in low-overlap three-speaker mixtures due to explicit separation constraints, H-SAGE demonstrates superior robustness and generalization under realistic, complex overlap.

Ablation and Architectural Analysis

Ablation experiments confirm that both the Overlap-Aware Loss and the holistic gating mechanism are indispensable for peak performance, especially in challenging scenarios. Removal of explicit overlap supervision leads to overfitting to simple mixture structures and impairs zero-shot generalization. Furthermore, distributing experts across both attention and feed-forward modules yields incremental gains compared to applying experts to either in isolation, particularly when integrated with the SA-Encoder’s strong global context modeling.

Additional experiments show optimal auxiliary loss weighting at $\lambda=3$, balancing guidance and ASR performance. Excessive auxiliary influence degrades transcription, while insufficient guidance weakens speaker disentanglement benefits.

Theoretical and Practical Implications

The research establishes that explicit, frame-level modeling of acoustic overlap, coupled with contextually integrated expert selection, provides significant advances in robust multi-talker ASR. The delineation between implicit and explicit acoustic state modeling is empirically bridged, suggesting a shift toward fully grounded, supervised router designs in future MoE-based speech processing architectures.

Practically, H-SAGE's architectural principles are extensible to a broader class of sequence modeling problems where dynamic selection among specialized modules must be sensitive to temporally structured context cues. The explicit disentangling of overlap and speaker activity states is likely to inspire extensions to other high-entropy sequence transduction tasks, such as multi-modal ASR or conversational diarization within end-to-end frameworks.

Conclusion

H-SAGE delivers a substantial step forward in MTASR, moving beyond implicit adaptation to explicit, context-aware expert arbitration. By combining a Speaker-Aware Global Encoder, Overlap-Aware Loss, and Holistic Gating Mechanism, the model achieves superior transcription and speaker disentanglement under complex overlap conditions and demonstrates robust zero-shot generalization to unseen speaker scenarios. These contributions define new methodological directions for the principled design of MoE-based architectures in multi-source sequence modeling.