SA-WavLM: Speaker-Aware Speech Processing

• SA-WavLM is a family of speaker-aware architectures that leverages self-supervised learning with explicit speaker embeddings to effectively handle mixture speech.
• It employs an extract–merge–predict framework incorporating conditional layer normalization, speaker merge blocks, and speaker shuffling to disambiguate overlapping voices.
• SA-WavLM achieves state-of-the-art results in speech separation, overlapping speech detection, and multi-speaker ASR, as demonstrated by significant gains in SDRi and F1 scores.

SA-WavLM denotes a family of speaker-aware architectures and self-supervised learning strategies specifically designed to address the challenges posed by mixture speech, i.e., acoustic mixtures involving two or more speakers, often with overlap. These methods extend the WavLM framework by incorporating explicit speaker-awareness, either in self-supervised pre-training for multi-speaker speech representation learning or in downstream tasks such as overlapping speech detection (OSD). SA-WavLM approaches integrate speaker-specific modeling via speaker embeddings and conditioning, cross-attention, merge blocks, and tailored pre-training curricula, achieving state-of-the-art performance in speech separation, diarization, extraction, overlapping region detection, and multi-speaker ASR (Lin et al., 2024, Sun et al., 29 May 2025).

1. Motivation and Model Overview

Existing self-supervised speech representation models (e.g., WavLM, wav2vec 2.0, HuBERT) predominantly train on single-speaker utterances or noisy backgrounds, limiting their ability to represent mixture speech, especially when multiple speakers overlap. In these models, mixture signals are either treated by designating a "primary" source and relegating others to background noise (as in WavLM) or by jointly encoding all speakers without explicit speaker indexing (Cocktail HuBERT). This leads to degraded performance in tasks requiring speaker disambiguation, notably under conditions of heavy overlap or equal-importance speakers.

SA-WavLM remedies these shortcomings through two principal model variants:

• A self-supervised pre-training pipeline for representation learning on mixture speech based on an "extract–merge–predict" principle (Lin et al., 2024).
• A speaker-aware downstream detection and classification stack for overlapping speech detection, leveraging progressive training, cross-attention, and multi-task learning (Sun et al., 29 May 2025).

Both variants explicitly inject speaker information into the model, enabling frame-wise or contextual representation disentanglement for each talker in a mixture.

2. Speaker-Aware Self-Supervised Pre-training (Extract–Merge–Predict)

SA-WavLM pre-training for mixture speech implements a three-stage architecture:

1. Extract: Each speaker's contextual representation is extracted from the mixture using a Speaker Adapted Transformer Encoder (SATE). SATE incorporates a Conditional Layer Normalization (CLN) scheme, parameterized by a fixed embedding for each target speaker, typically generated by a separate speaker-verification (SV) model (e.g., CAM++). The CLN modifies the standard normalization via affine transformation conditioned on the speaker embedding:

$$\mathrm{CLN}(X, e^k) = \frac{X - \mu(X)}{\sqrt{\sigma^2(X) + \epsilon}} \otimes [\gamma + \mathbf{w}(e^k)] + [\beta + \bm{\theta}(e^k)]$$

where $X$ is the feature matrix, $e^k$ the speaker embedding, and $\mathbf{w}, \bm{\theta}$ linear projections.

1. Merge: To model interaction between co-active speakers, the architecture concatenates the separate extracted contextual sequences and processes the result through a Speaker Merge Block (SMB)—a lightweight Transformer layer with a linear projection.
2. Predict: Masked prediction heads independently recover HuBERT-based pseudo-labels for each speaker’s clean reference at masked positions. The aggregate supervised loss is computed as:

$$\mathcal{L}_{\text{mask}} = \sum_{k \in \{a,b\}} \sum_{t \in O^k} -\log p(u^k_t | z^k_t)$$

with $O^k$ the time indices of masked frames for speaker $k$, and $u^k_t$ the corresponding HuBERT labels.

Crucially, only the extraction stage (CNN encoder + SATE) is retained for downstream tasks. During pre-training, speaker shuffling is employed to randomize the assignment of speaker embeddings to output heads and to simulate speaker absences, thereby training the model to be invariant to input order and capable of outputting silence when a target speaker is absent (Lin et al., 2024).

3. Speaker Shuffling and Robustness to Speaker Permutation

Speaker shuffling is a central strategy in SA-WavLM pre-training to induce invariance to speaker order and to handle missing speakers robustly. The order of input speaker embeddings is randomized with the corresponding target labels. In single-speaker or noisy scenarios, a pair is formed by combining the actual speaker embedding with either (a) a randomly sampled non-participating speaker embedding or (b) a trainable "null" embedding, the latter mapped to a silence token in the pseudo-label sequence. This ensures that the model learns both output channel exchangeability and explicit detection of inactive speakers, a necessary property for scaling to variable and unknown numbers of speakers in real-world data (Lin et al., 2024).

4. Downstream Architectures for Overlapping Speech Detection

For OSD, a Speaker-Aware WavLM architecture combines a WavLM-Large encoder backbone with a cross-attention module and a progressive two-decoder stack. The process includes:

• Feature Extraction: Input waveforms are encoded by WavLM-Large and, in parallel, converted to frame-level speaker embeddings by a CampPlus SV network.
• Speaker Attention: Frame-level speaker embeddings are fused into acoustic features by a single cross-attention block, making the features “speaker-aware”.
• Progressive Decoding: Decoding is split into VAD (voice activity detection) and OSD (overlap detection) branches, each implemented as Conformer stacks. Temporal masks generated from the VAD output modulate attention to speech- and non-speech frames, with the output of VAD serving as a gate for the OSD branch (Sun et al., 29 May 2025).

This yields high-performance OSD, benefitting from hierarchical learning of speech presence before overlap detection.

5. Pre-training, Implementation, and Evaluation Protocols

The SA-WavLM pre-training uses synthetic mixtures from LibriSpeech (960 h), with noise from the DNS dataset, covering clean and noisy single-speaker, as well as two-speaker overlap scenarios. Speaker embeddings are generated per utterance by the CAM++ system. The WavLM Base backbone is partially initialized from pretrained weights, with new parameters (CLN, SMB) initialized randomly. Masking follows WavLM conventions (10% frame masking, 80/10/10 time-masking). Training runs for 400k steps at a learning rate of $7 \times 10^{-5}$, using HuBERT 9th-layer clusters as targets and a speaker-shuffle probability $\alpha = 0.5$ (Lin et al., 2024).

For OSD, the model is trained in two phases: (1) VAD pretraining on simulated LibriHeavyMix, and (2) joint VAD+OSD fine-tuning on real meeting data (AliMeeting, AMI). Multi-task mean squared error loss is employed for both VAD and OSD, with balanced sampling to handle class imbalance. Metrics include frame-level recall, precision, and F1 score (Sun et al., 29 May 2025).

6. Experimental Results and Comparative Performance

SA-WavLM demonstrates state-of-the-art performance across several mixture-speech tasks. Benchmarks in SUPERB mixture-speech tasks, multi-speaker ASR, and speech extraction/separation are summarized below:

Model	SE PESQ↑	SE STOI↑	SS SI-SDRi↑	SD DER↓	ASR WER (w/LM)↓
WavLM Base	2.58	94.00	10.37	4.55	11.36
Cocktail HuBERT	2.63	94.00	11.08	2.77	–
SA-WavLM (Base)	2.62	94.18	11.13	1.88	6.49

• SA-WavLM yields +3.74 dB SDRi over strong BSRNN extraction features and consistently outperforms baseline HuBERT and WavLM in ConvTasNet separation with limited training data.
• In OSD, the SA-WavLM-based model achieves an F1 score of 82.76% on the AMI test set, surpassing XLSR-Conformer by 3.55 points. Ablations show that removing the speaker attention module or the progressive curriculum substantially degrades performance (Sun et al., 29 May 2025).

7. Limitations and Future Directions

While SA-WavLM achieves improvements in robustness and representational capacity for mixture speech, limitations remain. Current methods focus on two-speaker mixtures; extension to mixtures with three or more co-located talkers remains a challenge. Both the pre-training and OSD pipelines depend on external, fixed speaker embedding networks (e.g., CAM++, CampPlus)—joint or end-to-end learning of SV modules may further improve speaker representation consistency and downstream diarization. Future research directions include integration of contrastive objectives or permutation-invariant losses (e.g., PIT), domain adaptation to far-field and noisy conditions, and automatic task weighting strategies for multi-task learning (Lin et al., 2024, Sun et al., 29 May 2025).