---
title: Bi-label Masked Speech Prediction (MSP)
url: https://www.emergentmind.com/topics/bi-label-masked-speech-prediction-msp
type: topic
---

# Bi-label Masked Speech Prediction (MSP)

Bi-label Masked Speech Prediction (MSP) is an extension of self-supervised masked speech prediction that leverages multiple ground truth pseudo-label streams per frame, enabling the representation and prediction of overlapping acoustic sources. Designed to address the inherent limitations of conventional single-label MSP objectives in multi-speaker and acoustically complex conditions, bi-label MSP jointly models separable information streams—such as multiple speakers or competing background conditions—by training dual output heads. This approach underpins recent state-of-the-art results on streaming multi-talker automatic speech recognition (ASR), speech separation (SS), speech enhancement (SE), and controllable text-to-speech (TTS), as detailed in [2211.05564], [2211.06041], and [2502.07345].

## 1. Motivation and Conceptual Foundations

Conventional masked speech prediction, as operationalized by HuBERT or WavLM, treats utterance mixtures by predicting the pseudo-labels of only the dominant (primary) speaker per masked frame. While effective in single-speaker scenarios, this leads to representational degradation for non-dominant (secondary) sources and systematic biases against minority speakers, particularly in overlap regions. To rectify this, bi-label MSP requires simultaneous prediction of both primary and secondary pseudo-label streams for every masked frame within a mixture, enforcing parallel retention of multiple talkers’ representations [2211.05564], [2211.06041]. This paradigm generalizes to other acoustic "labels," such as clean and augmented background streams for background removal/preservation in TTS [2502.07345].

## 2. Mathematical Formulation

Let $x_{1:T}$ denote the sequence of input acoustic features, and $M \subset \{1, ..., T\}$ the indices of randomly chosen masked frames. For overlapping utterances, two pseudo-label sequences are constructed by applying frame-level quantizers to the unmixed reference tracks:

- $r^{pr}_{1:T}$: primary speaker/target label sequence
- $r^{sc}_{1:T}$: secondary speaker/target label sequence (assigned a blank symbol if absent at $t$)

After encoding with a CNN and L-layer transformer, two separate projection heads output logits $o^{pr}_t,o^{sc}_t \in\mathbb{R}^C$ for each masked $t \in M$. Each head produces posteriors via cosine-based scoring:
$$
p(c|o) = \frac{\exp(\cos(o \cdot W^P, e_c)/\gamma)}{\sum_{c'=1}^C \exp(\cos(o\cdot W^P,e_{c'})/\gamma)}
$$
The bi-label MSP loss is:
$$
L_{\mathrm{BiMSP}} = \sum_{t\in M} \Big( -\log p(r^{pr}_t | o^{pr}_t) - \log p(r^{sc}_t | o^{sc}_t) \Big)
$$
[2211.05564].

For speech separation and enhancement, a mathematically equivalent multi-label loss is used. For input mixtures yielding two code sequences $Z^1,Z^2 \in \{1,...,C\}^T$, the multi-head MSP loss is:
$$
L_{\mathrm{MSP}} = \sum_{t\in M} \big( -\log p^1(z^1_t|\mathbf{o}_t) - \log p^2(z^2_t|\mathbf{o}_t) \big)
$$
with cosine similarity-based posteriors [2211.06041]. Analogous constructs apply to bi-label controllable MSP for dual background removal/preservation in flow-matching TTS [2502.07345].

## 3. Architectural Extensions and Adapter Integration

The backbone for bi-label MSP typically follows HuBERT/WavLM's CNN-fronted transformer architecture. To support dual target streams:

- The final classification head is replaced by two parallel projection heads (one per label sequence) [2211.05564], [2211.06041].
- During training, only these heads and a subset of transformer layers are updated; the remainder (including the CNN) can remain frozen.
- For multi-label adaptation in speech separation/enhancement, lightweight bottleneck adapters are inserted into each transformer layer (post-attention and post-feedforward), with all original parameters frozen. These adapters comprise a layer norm, a down-projection, a ReLU, and an up-projection, and add marginal parameter cost (e.g., +3.1% for $k=64$) [2211.06041].
- In TTS applications, a dual-speaker encoder employs two separate transformers for distinct background tasks (removal/preservation), gated by a binary control signal and selectively fused into the main acoustic backbone [2502.07345].

These modifications ensure that information for each target stream is independently reconstructed and retained, while maintaining model efficiency and preventing interference with single-source tasks.

## 4. Quantizer, Masking, and Data Regimes

Pseudo-labels are derived from one of several frame-level quantization strategies:

- **FBANK Clustering**: K-means on mel-filterbank features (typically $C=500$ clusters).
- **HuBERT Embedding Clustering**: Clustering of hidden representations from a pre-trained HuBERT model.
- **Phoneme Quantizer**: Output sequences from a hybrid ASR, typically with 347 phonemes plus a blank symbol [2211.05564].

Masking follows HuBERT conventions: for ASR/SS/SE, mask spans (e.g., 8% of frames, each 10 frames long) are replaced by a learnable embedding [2211.05564]. For multi-label speech enhancement, up to 49% of frames may be masked [2211.06041]. In TTS, contiguous or block-random masks are sampled over the mel-spectrogram [2502.07345].

Pre-training data for bi-label MSP is generated through mixture and augmentation regimes:

| Task                | Mixture Types                                    | Ratios           | Datasets / Noise                     |
|---------------------|--------------------------------------------------|------------------|--------------------------------------|
| Streaming ASR       | 1-spk, 2-spk                                     | 50:50            | LibriSpeech 960h                     |
| SS/SE               | raw, noisy, 2-spk, noisy-overlapped              | 1:1:1:1          | LibriSpeech 960h + DNS, WHAM! (tot. $\sim$3840h) |
| TTS (background)    | clean, WHAM! noise, RIR, interfering speech, clean | 20:20:20:20:40   | LibriTTS 585h with augmentations     |

## 5. Training and Inference Procedures

### ASR/SS/SE Pre-training and Fine-tuning

- **Optimizers**: AdamW with linear decay and warmup; e.g., peak LR $1.5 \times 10^{-3}$ for ASR pre-training [2211.05564], Adam for SS/SE [2211.06041].
- **Batch Sizes**: ASR pre-training at 480 s/GPU (16 GPUs); SS/SE pre-training at 150–200 s/GPU (8 GPUs).
- **Fine-tuning**: ASR uses RNN-T with t-SOT style transducer; SS/SE uses BiLSTM decoders for PIT-MSE (for SS) or SI-SNR (for SE) losses [2211.06041].
- **Masking**: Masking proportions and randomization schedule as in pre-training.

### Streaming and TTS Inference

- **Streaming ASR**: Implements chunk-wise attention masks (e.g., chunk size 4 = 40 ms, 160 ms total latency). Outputs are serialized with interleaved speaker tokens; deserializers map sequences back to individual talker hypotheses [2211.05564].
- **Controllable TTS**: A phoneme duration predictor, a flow-matching model with bi-label CMSP loss, and a HiFiGAN vocoder are trained separately. The binary control signal toggles between background removal and preservation on-the-fly, with inference conducted via ODE integration over the flow-matching dynamics [2502.07345].

## 6. Empirical Results and Ablations

Bi-label MSP consistently achieves substantial improvements over conventional single-label MSP in multi-source scenarios, with especially strong gains observed in overlapped and noisy conditions.

### Automatic Speech Recognition (LibriSpeechMix, t-SOT, 160 ms latency) [2211.05564]:

| Objective/Quantizer | Dev1spk | Dev2spk | Test1spk | Test2spk |
|---------------------|---------|---------|----------|----------|
| No pre-train        | 15.42   | 39.12   | 15.69    | 39.52    |
| MSP FBANK           | 13.17   | 36.13   | 13.20    | 35.29    |
| Bi-label MSP FBANK  | 13.29   | 25.68   | 13.90    | 25.78    |
| MSP HuBERT          | 10.77   | 17.24   | 11.30    | 17.25    |
| Bi-label MSP HuBERT | 10.82   | 15.84   | 11.19    | 15.30    |
| MSP Phoneme         |  9.80   | 15.45   |  9.96    | 15.13    |
| Bi-label MSP Phoneme|  9.47   | 13.89   |  9.84    | 13.74    |

Bi-label MSP delivers a 26.9% relative reduction in two-speaker WER (e.g., 35.3%→25.8% with FBANK), with smaller gains in one-speaker WER. With phoneme quantization, two-speaker WER falls to 13.7% [2211.05564].

### Speech Separation/Enhancement/ASR (SUPERB-SG, frozen backbone) [2211.06041]:

| Method           | SS SI-SDR↑ | SE PESQ↑ | SE STOI↑ | ASR w/o LM↓ | ASR w/ LM↓ |
|------------------|------------|----------|----------|-------------|------------|
| Fbank baseline   | 9.15       | 1.84     | 87.25    | 26.36       | 17.37      |
| HuBERT Base      | 9.40       | 1.95     | 88.04    | 6.60        | 4.88       |
| Multi-label (no adapter) | 10.59 | 2.08 | 89.20 | 9.66        | 6.84       |
| Adapter256       | 10.17      | 2.04     | 89.04    | 6.80        | 5.02       |

Ablations confirm the benefit of two-head MSP: SI-SDR improves from 9.69 (data augment) to 10.59 (multi-label) [2211.06041].

### Controllable TTS (SIM/MCD/MOS/B-MOS) [2502.07345]:

- CMSP with dual encoder matches clean-speech performance for removal mode and achieves substantial MOS/B-MOS gains for background preservation.
- Dual encoders yield up to 0.05 SIM and 0.6 B-MOS improvement over standard MSD, especially in interfering speech scenarios.
- Speaker similarity remains robust across SNRs from –5 dB to +10 dB.

## 7. Significance, Implications, and Limitations

Bi-label MSP frameworks enforce preservation and explicit disentangling of information for multiple concurrent sources, notably improving model robustness in overlapping and acoustically challenging settings. For streaming ASR, the approach allows the transformer encoder to retain and transmit features for all concurrent speakers, directly benefiting transducer-based serialized output. In enhancement and separation, bi-label MSP enables learning of both separation and denoising functions in shared representations or lightweight adapter spaces, offering parameter efficiency and faster pre-training [2211.06041]. In TTS, bi-label CMSP with dual encoders provides user-controllable background manipulation without retraining [2502.07345].

Key limitations include the restriction to two-label regimes (i.e., extending to $N>2$ sources remains an open direction), potential conflicts between mixture-aware and single-source representation learning (mitigated by adapters), and imperfect handling of semantic mismatch in interfering speaker preservation [2211.06041], [2502.07345]. Promising future extensions involve curriculum schemes, dynamic loss weighting, integration of multi-speaker transcripts, and more parameter-efficient shared backbones across multiple tasks.

---

**Key References:**  
- "Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition" [2211.05564]  
- "An Adapter based Multi-label Pre-training for Speech Separation and Enhancement" [2211.06041]  
- "Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction" [2502.07345]

Source: https://www.emergentmind.com/topics/bi-label-masked-speech-prediction-msp