Bi-label Masked Speech Prediction (MSP)

• Bi-label MSP is an innovative self-supervised approach that predicts dual pseudo-label streams per frame to represent overlapping acoustic sources.
• It employs a dual projection head in a CNN-transformer backbone to separately model primary and secondary speaker signals in complex audio mixtures.
• Empirical results demonstrate significant gains, including up to a 26.9% reduction in two-speaker WER and improvements in speech separation and TTS performance.

Bi-label Masked Speech Prediction (MSP) is an extension of self-supervised masked speech prediction that leverages multiple ground truth pseudo-label streams per frame, enabling the representation and prediction of overlapping acoustic sources. Designed to address the inherent limitations of conventional single-label MSP objectives in multi-speaker and acoustically complex conditions, bi-label MSP jointly models separable information streams—such as multiple speakers or competing background conditions—by training dual output heads. This approach underpins recent state-of-the-art results on streaming multi-talker automatic speech recognition (ASR), speech separation (SS), speech enhancement (SE), and controllable text-to-speech (TTS), as detailed in (Huang et al., 2022, Wang et al., 2022), and (Zhang et al., 11 Feb 2025).

1. Motivation and Conceptual Foundations

Conventional masked speech prediction, as operationalized by HuBERT or WavLM, treats utterance mixtures by predicting the pseudo-labels of only the dominant (primary) speaker per masked frame. While effective in single-speaker scenarios, this leads to representational degradation for non-dominant (secondary) sources and systematic biases against minority speakers, particularly in overlap regions. To rectify this, bi-label MSP requires simultaneous prediction of both primary and secondary pseudo-label streams for every masked frame within a mixture, enforcing parallel retention of multiple talkers’ representations (Huang et al., 2022, Wang et al., 2022). This paradigm generalizes to other acoustic "labels," such as clean and augmented background streams for background removal/preservation in TTS (Zhang et al., 11 Feb 2025).

2. Mathematical Formulation

Let $x_{1:T}$ denote the sequence of input acoustic features, and $M \subset \{1, ..., T\}$ the indices of randomly chosen masked frames. For overlapping utterances, two pseudo-label sequences are constructed by applying frame-level quantizers to the unmixed reference tracks:

• $r^{pr}_{1:T}$: primary speaker/target label sequence
• $r^{sc}_{1:T}$: secondary speaker/target label sequence (assigned a blank symbol if absent at $t$)

After encoding with a CNN and L-layer transformer, two separate projection heads output logits $o^{pr}_t,o^{sc}_t \in\mathbb{R}^C$ for each masked $t \in M$. Each head produces posteriors via cosine-based scoring:

$$p(c|o) = \frac{\exp(\cos(o \cdot W^P, e_c)/\gamma)}{\sum_{c'=1}^C \exp(\cos(o\cdot W^P,e_{c'})/\gamma)}$$

The bi-label MSP loss is:

$$L_{\mathrm{BiMSP}} = \sum_{t\in M} \Big( -\log p(r^{pr}_t | o^{pr}_t) - \log p(r^{sc}_t | o^{sc}_t) \Big)$$

(Huang et al., 2022).

For speech separation and enhancement, a mathematically equivalent multi-label loss is used. For input mixtures yielding two code sequences $Z^1,Z^2 \in \{1,...,C\}^T$, the multi-head MSP loss is:

$M \subset \{1, ..., T\}$0

with cosine similarity-based posteriors (Wang et al., 2022). Analogous constructs apply to bi-label controllable MSP for dual background removal/preservation in flow-matching TTS (Zhang et al., 11 Feb 2025).

3. Architectural Extensions and Adapter Integration

The backbone for bi-label MSP typically follows HuBERT/WavLM's CNN-fronted transformer architecture. To support dual target streams:

• The final classification head is replaced by two parallel projection heads (one per label sequence) (Huang et al., 2022, Wang et al., 2022).
• During training, only these heads and a subset of transformer layers are updated; the remainder (including the CNN) can remain frozen.
• For multi-label adaptation in speech separation/enhancement, lightweight bottleneck adapters are inserted into each transformer layer (post-attention and post-feedforward), with all original parameters frozen. These adapters comprise a layer norm, a down-projection, a ReLU, and an up-projection, and add marginal parameter cost (e.g., +3.1% for $M \subset \{1, ..., T\}$1) (Wang et al., 2022).
• In TTS applications, a dual-speaker encoder employs two separate transformers for distinct background tasks (removal/preservation), gated by a binary control signal and selectively fused into the main acoustic backbone (Zhang et al., 11 Feb 2025).

These modifications ensure that information for each target stream is independently reconstructed and retained, while maintaining model efficiency and preventing interference with single-source tasks.

4. Quantizer, Masking, and Data Regimes

Pseudo-labels are derived from one of several frame-level quantization strategies:

• FBANK Clustering: K-means on mel-filterbank features (typically $M \subset \{1, ..., T\}$2 clusters).
• HuBERT Embedding Clustering: Clustering of hidden representations from a pre-trained HuBERT model.
• Phoneme Quantizer: Output sequences from a hybrid ASR, typically with 347 phonemes plus a blank symbol (Huang et al., 2022).

Masking follows HuBERT conventions: for ASR/SS/SE, mask spans (e.g., 8% of frames, each 10 frames long) are replaced by a learnable embedding (Huang et al., 2022). For multi-label speech enhancement, up to 49% of frames may be masked (Wang et al., 2022). In TTS, contiguous or block-random masks are sampled over the mel-spectrogram (Zhang et al., 11 Feb 2025).

Pre-training data for bi-label MSP is generated through mixture and augmentation regimes:

Task	Mixture Types	Ratios	Datasets / Noise
Streaming ASR	1-spk, 2-spk	50:50	LibriSpeech 960h
SS/SE	raw, noisy, 2-spk, noisy-overlapped	1:1:1:1	LibriSpeech 960h + DNS, WHAM! (tot. $M \subset \{1, ..., T\}$33840h)
TTS (background)	clean, WHAM! noise, RIR, interfering speech, clean	20:20:20:20:40	LibriTTS 585h with augmentations

5. Training and Inference Procedures

ASR/SS/SE Pre-training and Fine-tuning

• Optimizers: AdamW with linear decay and warmup; e.g., peak LR $M \subset \{1, ..., T\}$4 for ASR pre-training (Huang et al., 2022), Adam for SS/SE (Wang et al., 2022).
• Batch Sizes: ASR pre-training at 480 s/GPU (16 GPUs); SS/SE pre-training at 150–200 s/GPU (8 GPUs).
• Fine-tuning: ASR uses RNN-T with t-SOT style transducer; SS/SE uses BiLSTM decoders for PIT-MSE (for SS) or SI-SNR (for SE) losses (Wang et al., 2022).
• Masking: Masking proportions and randomization schedule as in pre-training.

Streaming and TTS Inference

• Streaming ASR: Implements chunk-wise attention masks (e.g., chunk size 4 = 40 ms, 160 ms total latency). Outputs are serialized with interleaved speaker tokens; deserializers map sequences back to individual talker hypotheses (Huang et al., 2022).
• Controllable TTS: A phoneme duration predictor, a flow-matching model with bi-label CMSP loss, and a HiFiGAN vocoder are trained separately. The binary control signal toggles between background removal and preservation on-the-fly, with inference conducted via ODE integration over the flow-matching dynamics (Zhang et al., 11 Feb 2025).

6. Empirical Results and Ablations

Bi-label MSP consistently achieves substantial improvements over conventional single-label MSP in multi-source scenarios, with especially strong gains observed in overlapped and noisy conditions.

Automatic Speech Recognition (LibriSpeechMix, t-SOT, 160 ms latency) (Huang et al., 2022):

Objective/Quantizer	Dev1spk	Dev2spk	Test1spk	Test2spk
No pre-train	15.42	39.12	15.69	39.52
MSP FBANK	13.17	36.13	13.20	35.29
Bi-label MSP FBANK	13.29	25.68	13.90	25.78
MSP HuBERT	10.77	17.24	11.30	17.25
Bi-label MSP HuBERT	10.82	15.84	11.19	15.30
MSP Phoneme	9.80	15.45	9.96	15.13
Bi-label MSP Phoneme	9.47	13.89	9.84	13.74

Bi-label MSP delivers a 26.9% relative reduction in two-speaker WER (e.g., 35.3%→25.8% with FBANK), with smaller gains in one-speaker WER. With phoneme quantization, two-speaker WER falls to 13.7% (Huang et al., 2022).

Speech Separation/Enhancement/ASR (SUPERB-SG, frozen backbone) (Wang et al., 2022):

Method	SS SI-SDR↑	SE PESQ↑	SE STOI↑	ASR w/o LM↓	ASR w/ LM↓
Fbank baseline	9.15	1.84	87.25	26.36	17.37
HuBERT Base	9.40	1.95	88.04	6.60	4.88
Multi-label (no adapter)	10.59	2.08	89.20	9.66	6.84
Adapter256	10.17	2.04	89.04	6.80	5.02

Ablations confirm the benefit of two-head MSP: SI-SDR improves from 9.69 (data augment) to 10.59 (multi-label) (Wang et al., 2022).

Controllable TTS (SIM/MCD/MOS/B-MOS) (Zhang et al., 11 Feb 2025):

• CMSP with dual encoder matches clean-speech performance for removal mode and achieves substantial MOS/B-MOS gains for background preservation.
• Dual encoders yield up to 0.05 SIM and 0.6 B-MOS improvement over standard MSD, especially in interfering speech scenarios.
• Speaker similarity remains robust across SNRs from –5 dB to +10 dB.

7. Significance, Implications, and Limitations

Bi-label MSP frameworks enforce preservation and explicit disentangling of information for multiple concurrent sources, notably improving model robustness in overlapping and acoustically challenging settings. For streaming ASR, the approach allows the transformer encoder to retain and transmit features for all concurrent speakers, directly benefiting transducer-based serialized output. In enhancement and separation, bi-label MSP enables learning of both separation and denoising functions in shared representations or lightweight adapter spaces, offering parameter efficiency and faster pre-training (Wang et al., 2022). In TTS, bi-label CMSP with dual encoders provides user-controllable background manipulation without retraining (Zhang et al., 11 Feb 2025).

Key limitations include the restriction to two-label regimes (i.e., extending to $M \subset \{1, ..., T\}$5 sources remains an open direction), potential conflicts between mixture-aware and single-source representation learning (mitigated by adapters), and imperfect handling of semantic mismatch in interfering speaker preservation (Wang et al., 2022, Zhang et al., 11 Feb 2025). Promising future extensions involve curriculum schemes, dynamic loss weighting, integration of multi-speaker transcripts, and more parameter-efficient shared backbones across multiple tasks.

---

Key References:

• "Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition" (Huang et al., 2022)
• "An Adapter based Multi-label Pre-training for Speech Separation and Enhancement" (Wang et al., 2022)
• "Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction" (Zhang et al., 11 Feb 2025)