Speaker Consistency-Aware Target Speaker Extraction

• The paper introduces a centroid-based speaker consistency loss that robustly aligns the extracted vocal signal with the enrolled speaker, reducing errors like speaker confusion.
• It employs a conditional loss suppression mechanism to balance identity alignment with separation quality, preventing overfitting in highly overlapping conditions.
• Experimental evaluations on Libri2Mix and cross-domain datasets demonstrate measurable improvements in SI-SDR and speaker similarity, benefiting applications like verification and ASR.

Speaker consistency-aware target speaker extraction refers to a class of algorithms and system designs that enhance the fidelity and reliability of extracting a target speaker’s speech from a noisy or multi-talker mixture by explicitly maximizing the similarity—i.e., the consistency—between the speaker characteristics of the reference (enroLLMent) speech and those of the extracted output. Unlike conventional target speaker extraction (TSE) approaches that focus primarily on separation or denoising accuracy, speaker consistency-aware methods add dedicated mechanisms and objectives to ensure that the output remains true in identity to the target speaker, thereby reducing errors such as speaker confusion, cross-talk, or identity leakage, which can severely degrade intelligibility and downstream usability in applications such as speaker verification, transcription, and voice-based security.

1. Speaker Consistency: Motivation and Definition

Speaker consistency in TSE describes the degree to which the extracted speech matches the enrolled (reference) speaker in identity, despite possible variation in spoken content. This is distinct from general speech quality: the goal is not only to recover clean, intelligible speech but specifically to guarantee that all extracted segments “sound like” the target speaker. Achieving this ensures robust support for downstream tasks such as speaker verification, diarization, and any process that depends on persistent voice identity.

A core problem motivating speaker consistency-aware techniques is speaker identity confusion. In challenging mixtures—those with similar-sounding speakers, high overlap, or noise—the extracted speech may inadvertently pick up traits, or even entire segments, from non-target voices. This risk is particularly high when speaker embeddings (extracted from pre-trained encoders or models trained for verification) lack sufficient discriminative capacity in open-domain TSE settings (Wu et al., 13 Jul 2025). Ensuring that the extracted and reference speech share the same speaker identity is vital for reliable speaker-aware processing.

2. Centroid-Based Speaker Consistency Loss

The central contribution of SC-TSE is the introduction of a centroid-based speaker consistency loss to the TSE framework (Wu et al., 13 Jul 2025). In a standard TSE pipeline, a speaker encoder generates an embedding from both the enroLLMent speech and the output of the separator (e.g., a Band-Split RNN). These embeddings are then compared to assess consistency.

Instead of simply using the enrolled speech embedding, SC-TSE computes a “speaker centroid” for each speaker by averaging utterance-level embeddings across that speaker’s data:

$$\mathbf{e}_i^{C} = \frac{1}{K} \sum_{k=1}^{K} \mathbf{e}_{i,k}^{U}$$

where $\mathbf{e}_{i,k}^{U}$ is the embedding for utterance $k$ of speaker $i$. This centroid represents a robust, speaker-level identity reference.

The centroid-based consistency loss is formulated as:

$$L_{C\text{-}SC} = -\log \left( \frac{\exp(\cos(\hat{\mathbf{e}}_s, \mathbf{e}_{iT}^{C}))}{\sum_{i=1}^N \exp(\cos(\hat{\mathbf{e}}_s, \mathbf{e}_i^{C}))} \right)$$

where $\hat{\mathbf{e}}_s$ is the embedding of the separated signal, $\mathbf{e}_{iT}^{C}$ is the centroid of the current target speaker, and $N$ is the number of speakers. This loss acts as a softmax-based cross-entropy over all centroids, directly improving the discriminative alignment between extracted output and the intended speaker.

This approach differs from standard speaker consistency losses, which typically enforce direct cosine similarity between the enroLLMent and output embeddings:

$$L_{SC} = 1 - \cos(\mathbf{e}_r, \hat{\mathbf{e}}_s)$$

The centroid-based formulation is more robust to variability across utterances and mitigates speaker confusion more effectively.

3. Conditional Loss Suppression Mechanism

A practical observation is that optimizing for speaker consistency too aggressively can be detrimental once high similarity is reached: further pressure can cause overfitting, degrade separation quality, or even lead to unwanted distortions in the extracted signal. To counteract this, SC-TSE incorporates conditional loss suppression (CLS).

CLS applies a threshold-driven suppression to the centroid-based loss:

$$f_C(x) = \begin{cases} x & \text{if SECS} \leq \omega \ 0 & \text{otherwise} \end{cases}$$

with speaker encoder cosine similarity (SECS) assessed for each batch. The threshold $\omega$ is scheduled to decrease from 1.0 to 0.8 during training. This ensures the consistency loss is only active when necessary, preventing excessive focus on speaker alignment at the cost of other signal qualities.

The final loss function is a weighted combination:

$$L = (1 - \beta - \lambda) L_{SI\text{-}SDR} + \beta L_{CE} + f_C(\lambda L_{C\text{-}SC})$$

where $L_{SI\text{-}SDR}$ is the main separation loss, $L_{CE}$ is an optional classification loss, and $\lambda, \beta$ control the influence of each term.

4. Experimental Design and Evaluation Metrics

The SC-TSE methodology is evaluated using the Libri2Mix corpus (two-speaker, fully overlapped, 16 kHz mixtures). Key experimental ingredients include:

• Use of ECAPA-TDNN or ResNet34 speaker encoders, tested in both pretrained and joint training modes.
• Use of Band-Split RNN, DPCCN, and TF-GridNet as separator backbones, evaluating generality.
• Main metrics:
  • SI-SDR (Scale-Invariant Signal-to-Distortion Ratio)
  • SDR (Signal-to-Distortion Ratio)
  • PESQ (Perceptual Evaluation of Speech Quality)
  • STOI (Short-Time Objective Intelligibility)
  • Accuracy (fraction of samples with SI-SDR improvement > 1 dB)
  • Speaker similarity (cosine similarity between enroLLMent and extracted speech embeddings).

Integration of centroid-based consistency loss and CLS consistently improves SI-SDR (e.g., +0.51 dB with ECAPA-TDNN pretrained) and speaker similarity (e.g., +2.64%), with similar or improved outcomes across architectures and test domains.

SC-TSE demonstrates strong generalization, maintaining or improving performance with different separator architectures and when tested cross-domain on datasets such as Aishell2Mix and VoxCeleb1, with reduced speaker confusion rates.

5. Technical Implementation and Key Formulas

Some representative equations used in SC-TSE include:

• Speaker encoder cosine similarity (SECS):

$$SECS = \cos(\mathbf{e}_r, \hat{\mathbf{e}}_s) = \cos(E_\theta(r), E_\theta(\hat{s}))$$

• Utterance to centroid embedding:

$\mathbf{e}_{i,k}^{U} = E_\theta(x_{i,k})$

$$\mathbf{e}_i^{C} = \frac{1}{K} \sum_{k=1}^{K} \mathbf{e}_{i,k}^{U}$$

• Centroid-based loss with CLS:

$$L = (1 - \beta - \lambda) L_{SI\text{-}SDR} + \beta L_{CE} + f_C(\lambda L_{C\text{-}SC})$$

All training and evaluation were conducted using Libri2Mix, with validation on both the primary set and on cross-domain corpora for robustness.

6. Broader Impact and Future Directions

This work positions speaker consistency as an explicit optimization target—distinct from general separation or denoising—in TSE system design. By introducing centroid-derived loss functions and conditional suppression, the methodology enables robust, identity-faithful extraction even under mismatched, open-domain, or highly overlapped conditions.

Downstream benefits include:

• Reduction in speaker confusion and improved reliability for speaker-aware ASR and verification.
• Enhanced performance for real-world tasks such as meeting diarization, transcription, and voice-based authentication, where persistence of speaker identity is critical.

Future directions mentioned include extending consistency-aware training to other separation backbone architectures, further exploring alternative strategies for speaker consistency, and integrating recent advances in self-supervised learning or speaker modeling to improve generalization.

7. Conclusion

Speaker consistency-aware target speaker extraction incorporates identity-preserving objectives into the TSE pipeline. The centroid-based loss—augmented with conditional suppression—maximizes the alignment between enroLLMent and extracted speaker identity, and experimental results confirm consistent improvements in both separation quality and speaker similarity. This approach demonstrates clear relevance for robust, real-world TSE deployments and provides a foundation for continued advancement in both algorithmic capability and application reliability (Wu et al., 13 Jul 2025).