H-SAGE: Speaker-Aware Experts for Multi-Talker ASR

• H-SAGE is a Mixture-of-Experts framework for multi-talker ASR that integrates a Speaker-Aware Global Encoder, Overlap-Aware Loss, and Holistic Gating to improve expert routing under high-overlap conditions.
• It addresses the cocktail party problem by fusing long-range speaker context with local acoustic cues, effectively mitigating temporal myopia and disambiguating overlapping speech.
• Empirical results on LibriSpeechMix show that H-SAGE outperforms baselines across low, mid, and high overlap settings, confirming its robustness in complex multi-speaker scenarios.

Searching arXiv for the cited H-SAGE paper and closely related context papers to ground the article in current literature. Holistic Speaker-Aware Guided Experts (H-SAGE) is a Mixture-of-Experts (MoE) framework for multi-talker automatic speech recognition (MTASR) that augments a Single-Input Single-Output (SISO) ASR backbone with explicit acoustic supervision and globally informed expert routing. In its original formulation, introduced in "H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR" (Guo et al., 2 Jul 2026), the method addresses overlapping-speech recognition by combining a Speaker-Aware Global Encoder, an auxiliary Overlap-Aware Loss, and a Holistic Gating Mechanism. The central design claim is that explicit acoustic guidance and globally informed routing improve expert collaboration, particularly under complex high-overlap conditions.

1. Problem setting and motivating deficiencies

Multi-talker ASR targets the "cocktail party problem": given a single-channel mixture with spontaneous turn-taking and frequent overlaps, the system must disentangle and transcribe utterances from multiple speakers. High-overlap segments are particularly challenging because phonetic and speaker cues are heavily entangled over time, and the attention-based serialization used in SISO models, such as Serialized Output Training (SOT), must implicitly track speakers without explicit guidance (Guo et al., 2 Jul 2026).

The H-SAGE paper identifies three limitations in recent MoE approaches for MTASR, including GLAD. First, frame-independent routing induces temporal myopia, because global signals come from a static projection and cannot model speaker activity dynamics. Second, reliance solely on the downstream ASR objective yields implicit and ungrounded representations, so routers learn shortcuts rather than robust acoustic states. Third, expert arbitration is driven essentially by local features, depriving the router of a holistic view. H-SAGE is therefore positioned not as a generic sparse MoE variant, but as a response to a specific failure mode in MoE-based MTASR: expert selection that lacks temporally informed, speaker-aware grounding (Guo et al., 2 Jul 2026).

The core hypothesis is stated in two parts: if the model learns speaker-aware global context with long-range temporal dependencies under an explicit overlap-aware supervision, and if it fuses that global context with local frame-level details to gate experts holistically, then MoE-based MTASR becomes significantly more robust, especially under high-overlap conditions. This suggests that H-SAGE reframes routing from a purely optimization-driven latent partition into a supervised acoustic-state arbitration problem.

2. Architectural composition

H-SAGE augments a MoE-based SISO ASR backbone with two modules designed to resolve the limitations above: a Speaker-Aware Global Encoder (SA-Encoder) and a Holistic Gating Mechanism. The backbone is a Conformer encoder paired with a 6-block Transformer decoder. The encoder is macaron-style with 4-head self-attention and 256 hidden units; encoder feed-forward dimensionality is 1024. The decoder has 4 heads, 256 hidden units, and 2048-dimensional feed-forward layers. The decoder performs SOT: given $N$ speakers, it produces a serialized sequence $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$, where $\langle sc\rangle$ is a speaker-change token and FIFO ordering is used (Guo et al., 2 Jul 2026).

The convolutional frontend extracts $X_{\text{speech}} \in \mathbb{R}^{T \times D}$. Each Conformer encoder block replaces its linear transformations with Mixture-of-Low-Rank-Experts (MoLE) layers. Local hidden states at each MoLE site provide $P_{\text{local}}$, the per-frame local routing distributions, as in GLAD/Local MoLE. In parallel, the SA-Encoder operates directly on $X_{\text{speech}}$ to produce $X_{\text{global}} \in \mathbb{R}^{T \times D}$ via multi-head self-attention and a feed-forward network, along with a framewise global router $$P_{\text{global}} = \text{softmax}(W_g X_{\text{global}} + b_g)$$. Crucially, $X_{\text{global}}$ is trained with an explicit Overlap-Aware Loss so that the global branch becomes speaker-aware and temporally informed (Guo et al., 2 Jul 2026).

At each time step, the Holistic Gating Mechanism concatenates local features $X_{\text{local}}$ and $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$0, computes fusion weights $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$1 through a small linear layer followed by softmax, and combines $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$2 and $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$3 into the final expert activation probabilities. The resulting routing is frame-dependent yet globally informed. Compared with frame-independent routing, the fusion mitigates temporal myopia; compared with purely local routing, it prevents information asymmetry and yields a comprehensive arbitration of experts (Guo et al., 2 Jul 2026).

A compact view of the architecture is as follows:

Component	Role	Output
Convolutional frontend	Extract early acoustic features	$$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$4
SA-Encoder	Model long-range speaker activity dynamics	$$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$5
MoLE-enhanced Conformer	Provide local framewise expert signals	$$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$6 and encoder states
Holistic Gating Mechanism	Fuse local and global routing evidence	Final expert activation probabilities
Transformer decoder with SOT	Decode serialized multi-speaker text	$$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$7

A common misconception is to reduce H-SAGE to the presence of low-rank experts alone. In the original system, the defining contribution is the coupling of those experts to explicit overlap-aware supervision and globally informed routing, rather than the MoLE parameterization by itself.

3. Speaker-aware supervision and routing formulation

The SA-Encoder is designed to overcome temporal myopia by modeling long-range dependencies in speaker activity. It operates on early acoustic features $$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$8, preserving rich low-level cues, and computes

$$Y = \{\text{text}_1, \langle sc\rangle, \text{text}_2, \ldots, \langle sc\rangle, \text{text}_N\}$$9

$\langle sc\rangle$0

with $\langle sc\rangle$1 for $\langle sc\rangle$2 experts (Guo et al., 2 Jul 2026).

The global branch is supervised by frame-level overlap-state labels derived from mixture construction, where $\langle sc\rangle$3 and $\langle sc\rangle$4 denotes padding, single-speaker, and overlapped speech states. A classification head on $\langle sc\rangle$5 yields $\langle sc\rangle$6 over classes $\langle sc\rangle$7, and the Overlap-Aware Loss is

$\langle sc\rangle$8

By optimizing $\langle sc\rangle$9, $X_{\text{speech}} \in \mathbb{R}^{T \times D}$0 encodes speaker activity dynamics and becomes a globally informative, speaker-aware signal that grounds routing decisions (Guo et al., 2 Jul 2026).

The Holistic Gating Mechanism combines local and global evidence by first forming

$X_{\text{speech}} \in \mathbb{R}^{T \times D}$1

$X_{\text{speech}} \in \mathbb{R}^{T \times D}$2

where $X_{\text{speech}} \in \mathbb{R}^{T \times D}$3 and $X_{\text{speech}} \in \mathbb{R}^{T \times D}$4. If $X_{\text{speech}} \in \mathbb{R}^{T \times D}$5 are the local and global coefficients, the final expert selection probabilities are

$X_{\text{speech}} \in \mathbb{R}^{T \times D}$6

This routing is frame-dependent but globally informed: decisions at each time step consider both the instantaneous local representation and the long-range, speaker-aware context (Guo et al., 2 Jul 2026).

Each MoLE layer follows a LoRA-style transformation:

$X_{\text{speech}} \in \mathbb{R}^{T \times D}$7

where $X_{\text{speech}} \in \mathbb{R}^{T \times D}$8 and $X_{\text{speech}} \in \mathbb{R}^{T \times D}$9 are shared linear parameters; expert $P_{\text{local}}$0 has low-rank matrices $P_{\text{local}}$1 and $P_{\text{local}}$2 with rank $P_{\text{local}}$3, scaled by $P_{\text{local}}$4; and $P_{\text{local}}$5 are the holistic expert activation probabilities (Guo et al., 2 Jul 2026).

Training combines recognition and explicit acoustic supervision. The ASR head uses standard cross-entropy on serialized SOT targets, $P_{\text{local}}$6, and the total objective is

$P_{\text{local}}$7

All components, including the SA-Encoder, the holistic gate $P_{\text{local}}$8, and expert parameters, are trained end-to-end; gradients flow through $P_{\text{local}}$9, and MoLE to jointly optimize separation and transcription. This suggests that H-SAGE treats routing, acoustic-state prediction, and recognition as a coupled optimization rather than as separable subproblems.

4. Experimental protocol and empirical performance

Experiments are conducted on LibriSpeechMix (LSM), derived from LibriSpeech, with 2-speaker and 3-speaker tests. The training set is constructed from single-speaker LibriSpeech plus synthesized 2-speaker mixtures, stratified by overlap rate into Low $X_{\text{speech}}$0, Mid $X_{\text{speech}}$1, and High $X_{\text{speech}}$2. Training statistics are 202,493 single-speaker utterances (692.2 h); 2-speaker Low/Mid/High: 81,808/75,245/45,423 utterances (491.8/383.0/203.2 h); total 404,985 utterances (1,770.2 h). Evaluation uses WER for single-speaker, Permutation-Invariant WER (PI-WER) for multi-speaker, and Overlap-Aware WER (OA-WER) averaged across Low/Mid/High strata. Models are trained on single- and 2-speaker data, and 3-speaker tests assess zero-shot generalization (Guo et al., 2 Jul 2026).

Implementation uses ESPnet2. To equalize parameters across baselines, encoder depth is set to 14 for SOT, 13 for SOT-SACTC, and 12 for SOT+Local MoLE, GLAD-SOT, and H-SAGE. H-SAGE uses $X_{\text{speech}}$3 experts per MoLE, with rank $X_{\text{speech}}$4 and scaling $X_{\text{speech}}$5. The auxiliary weight is $X_{\text{speech}}$6. Models are trained for 35 epochs on $X_{\text{speech}}$7 RTX 3090 GPUs using Adam with peak learning rate $X_{\text{speech}}$8 and 25,000 warm-up steps (Guo et al., 2 Jul 2026).

The main empirical finding is that H-SAGE consistently improves over strong baselines, with the largest gains in complex, high-overlap regimes.

Setting	H-SAGE	Comparison reported in the paper
Single-speaker LibriSpeech WER%	3.6 (Dev) / 3.8 (Test)	on par with SOT-SACTC 3.6/3.8 and GLAD-SOT 3.5/3.9
LSM-2mix overall PI-WER%	6.0 (Dev) / 5.7 (Test)	better than GLAD-SOT 6.0/6.2 and SOT-SACTC 6.9/6.7
LSM-3mix zero-shot overall PI-WER%	19.7 (Dev) / 19.5 (Test)	surpassing GLAD-SOT 19.9/19.8 and SOT-SACTC 20.6/20.0

For LSM-2mix Test under conditional evaluation, H-SAGE achieves 4.7/5.7/8.2/6.2 for Low/Mid/High/OA-WER, compared with GLAD-SOT at 5.1/6.4/8.9/6.8. The absolute improvements over GLAD-SOT are $X_{\text{speech}}$9 overall Test and $X_{\text{global}} \in \mathbb{R}^{T \times D}$0 at High overlap, with relative reductions of approximately $X_{\text{global}} \in \mathbb{R}^{T \times D}$1 overall and approximately $X_{\text{global}} \in \mathbb{R}^{T \times D}$2 at High overlap. For LSM-3mix zero-shot Test, H-SAGE records 15.5/18.9/24.9/19.8 for Low/Mid/High/OA-WER, versus GLAD-SOT at 15.0/19.5/25.5/20.0; the biggest gains are at Mid and High overlap, including $X_{\text{global}} \in \mathbb{R}^{T \times D}$3 absolute at High (Guo et al., 2 Jul 2026).

These results support a narrow but important interpretation: H-SAGE does not primarily improve already easy cases. Its gains concentrate in the exact regimes where implicit attention or frame-independent projections struggle, namely Mid/High overlap and 3-speaker zero-shot conditions.

5. Ablations and architectural interpretation

Component-wise studies quantify the contribution of explicit supervision, long-range global modeling, and holistic gating. Removing explicit supervision, denoted $X_{\text{global}} \in \mathbb{R}^{T \times D}$4, yields 3.5/3.7 on LibriSpeech, 6.1/5.8 on LSM-2mix, and 20.2/20.1 on LSM-3mix. The full H-SAGE is better on multi-speaker sets, especially 3-mix zero-shot, confirming the role of $X_{\text{global}} \in \mathbb{R}^{T \times D}$5 in generalization and overlap robustness (Guo et al., 2 Jul 2026).

Degenerating the global branch to a frame-independent projection, denoted $X_{\text{global}} \in \mathbb{R}^{T \times D}$6, gives 3.6/4.0, 6.2/6.4, and 20.8/21.5. The losses across all multi-speaker metrics highlight the necessity of long-range temporal modeling. Removing holistic gating and reverting to GLAD-style fusion yields 3.4/3.9, 6.0/6.2, and 20.4/20.9. H-SAGE’s holistic gating consistently outperforms, especially on multi-speaker tests, showing that globally informed routing is indispensable under overlap (Guo et al., 2 Jul 2026).

Expert placement studies further differentiate where low-rank specialization is most effective. Using experts only in FFN gives 3.9/4.2, 6.8/6.4, and 21.2/21.3; only in attention gives 3.3/3.7, 6.3/6.0, and 20.2/20.3; full H-SAGE, with experts in both, is best. With a strong SA-Encoder, attention-side experts are particularly effective, indicating structural compatibility between global cues and sequence modeling (Guo et al., 2 Jul 2026).

The paper’s interpretive claim is that explicit acoustic guidance enhances expert collaboration by grounding global context in speaker activity states. $X_{\text{global}} \in \mathbb{R}^{T \times D}$7 shapes $X_{\text{global}} \in \mathbb{R}^{T \times D}$8 and $X_{\text{global}} \in \mathbb{R}^{T \times D}$9 into informative signals that disambiguate single-speaker and multi-speaker regions, reducing router shortcut learning. Holistic gating mitigates temporal myopia by making framewise routing aware of long-range dynamics, improving specialization where overlap-induced entanglement is greatest. A plausible implication is that H-SAGE’s improvements arise less from increased parameter count than from better conditioning of routing decisions.

6. Efficiency, limitations, and broader usage of the H-SAGE concept

Parameter parity is enforced across systems: H-SAGE has 35.75M parameters, comparable to GLAD-SOT at 35.18M and smaller than vanilla SOT at 36.07M. The SA-Encoder adds a single attention+FFN global branch acting on $$P_{\text{global}} = \text{softmax}(W_g X_{\text{global}} + b_g)$$0, and the holistic gate is a light linear+softmax; training uses $$P_{\text{global}} = \text{softmax}(W_g X_{\text{global}} + b_g)$$1 RTX3090 GPUs with the same schedule. Latency and FLOPs overhead beyond parameter counts are not explicitly reported, but the design replaces existing linear layers with low-rank experts and adds lightweight global and gating modules, yielding practical scalability with three experts (Guo et al., 2 Jul 2026).

The principal limitations are also explicit. H-SAGE is trained on single- and 2-speaker data and then evaluated zero-shot on 3-speaker mixtures; while results are strong, Low-overlap 3-mix shows slight underperformance versus SACTC, suggesting that explicit separation constraints can help sparse overlap regimes. The overlap-aware supervision relies on synthetic mixture timing, specifically known delays and durations; inaccuracies in real-world boundaries or domain shifts could degrade $$P_{\text{global}} = \text{softmax}(W_g X_{\text{global}} + b_g)$$2 quality. Proposed future directions include integrating diarization-aware labels to refine supervision, extending training to variable speaker counts and cross-corpus data, and exploring more advanced global modeling such as hierarchical or recurrent global encoders and routing regularizers (Guo et al., 2 Jul 2026).

The term H-SAGE also appears in broader speaker-centric research as an organizing framework rather than solely as the original MTASR architecture. In "SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning" (Nam et al., 14 May 2026), H-SAGE is used as an interpretation of expert roles for profiling, condition analysis, comparison, and reasoning in a speaker-specialized audio-LLM. In "HumanOmni-Speaker: Identifying Who said What and When" (Bai et al., 23 Mar 2026), H-SAGE is presented as a way of structuring speaker-aware expert routing around audio-visual identity binding and Visual-Registered Speaker Diarization and Recognition. In "Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach" (Wei et al., 19 Jun 2026), the concept is connected to domain-aware MoE routing and proposed extensions toward speaker-aware and hierarchical gating. These uses do not replace the original MTASR definition, but they indicate that H-SAGE has rapidly become a transferable design vocabulary for speaker-aware expert systems across speech, audio-LLMs, audio-visual conversational understanding, and non-verbal speaker verification.

Reproducibility is comparatively strong. The original implementation uses ESPnet2, specifies the backbone, expert configuration, objective, and training schedule in detail, and provides code at the stated repository. Within the MTASR literature, this makes H-SAGE a concretely instantiated method rather than a purely conceptual proposal (Guo et al., 2 Jul 2026).