WSJ0-2Mix: Two-Speaker Separation Benchmark

• WSJ0-2Mix is a benchmark dataset for two-speaker overlapped-speech research derived from the Wall Street Journal corpus, emphasizing controlled conditions.
• It underpins advanced techniques like permutation invariant training, deep clustering, and Transformer-based methods, achieving substantial SDRi improvements.
• Extensions enable target speaker extraction, multi-speaker ASR, and dual-channel spatial separation, though critiques note limitations in diversity and real-world overlap.

WSJ0-2Mix is a benchmark dataset and experimental regime for two-speaker overlapped-speech research derived from the Wall Street Journal corpus. It is widely described as the de facto reference dataset for single-channel speech separation, while also serving as a common substrate for target speaker extraction, target-speaker automatic speech recognition, multi-speaker ASR, unknown-speaker-count separation, and dual-channel spatialized separation variants. Across this literature, WSJ0-2Mix functions less as a single immutable corpus than as a family of closely related recipes centered on two-speaker mixtures, usually at 8 kHz and often treated as fully overlapping or nearly fully overlapping speech (Yu et al., 2016, Cosentino et al., 2020).

1. Canonical dataset formulation

In its canonical use, WSJ0-2Mix is a single-channel, two-talker mixture benchmark constructed from the WSJ0 corpus. One standard summary reports training, validation, and test partitions of 30, 10, and 5 hours, corresponding to 20k, 5k, and 3k utterances at 8 kHz; related task-specific summaries report the same 20,000/5,000/3,000 split for target speaker extraction and target-speaker ASR derivatives of the benchmark (Subakan et al., 2022, Liu et al., 2023, Guo et al., 2024).

The benchmark is commonly described as a fully overlapping separation task. One ASR-oriented analysis characterizes it as simulated cross-talk where the speech of multiple speakers overlaps for almost the entire utterance, while later dataset critiques state more directly that WSJ0-2Mix mixtures have 100% overlap. The literature also distinguishes “min” and “max” subsets, with the former used for separation and counting in some studies and the latter used for ASR evaluation on full utterances (Menne et al., 2019, Neumann et al., 2020, Cosentino et al., 2020).

Several task-specific extensions preserve the WSJ0-2Mix mixture recipe while adding auxiliary structure. The target-speaker extraction version adds an enrollment utterance for the speaker of interest, with one summary reporting train and validation data from 101 speakers and a test set from 18 disjoint speakers, and an average enrollment duration of 7.3 s in the test set (Liu et al., 2023). WSJ0-2mix-extr adapts the benchmark to target-speaker ASR by pairing each mixture with a separate clean auxiliary utterance from the target speaker (Zhang et al., 2023). Dual-channel spatialized versions introduce source location labels and two-channel mixtures; one WSJ0-2Mix setup for SDNet uses two channels, 101 training and 18 test speakers, and source location labels at 5° increments (Li et al., 2021).

2. Benchmark role in the development of speaker-independent separation

WSJ0-2Mix became central because it exposed the label permutation problem in speaker-independent multi-talker speech separation. If a network produces $S$ outputs for $S$ sources, no fixed source-to-output ordering is available across mixtures. Permutation Invariant Training (PIT) addressed this by minimizing the separation loss over all assignments:

$$J_{\text{PIT}}=\min_{\pi \in \mathcal{P}} \frac{1}{T \times F \times S}\sum_{s=1}^{S}\left\| |\tilde{\mathbf{X}}_s|-|\mathbf{X}_{\pi(s)}| \right\|^2.$$

This formulation made speaker-independent regression feasible without predefined output identities and helped establish WSJ0-2Mix as the benchmark foundation for later separation work (Yu et al., 2016).

A parallel line of work used deep clustering rather than direct regression, learning embeddings for time-frequency bins and then clustering them into speaker-specific masks. In ASR preprocessing studies, deep clustering on WSJ0-2Mix was paired with a hybrid DNN-HMM recognizer, demonstrating that separation and recognition could be evaluated on the same benchmark rather than in isolation (Menne et al., 2019).

Later work treated the benchmark as a testbed not only for architectural innovation but also for alternative label-assignment strategies. “Interrupted and cascaded permutation invariant training” argued that standard PIT’s epoch-to-epoch assignment switching could be stabilized by inserting fixed-label phases derived from PIT itself, improving SDRi without modifying the base architecture (Yang et al., 2019). In a different direction, recursive separation with one-and-rest PIT (OR-PIT) reframed the task so that a single model could recursively separate one speaker and the residual mixture, rather than assuming a fixed number of outputs in advance:

$$L = \min_i~ l(\hat{s}(t), s_i(t)) + \frac{1}{N-1} l(\hat{r}(t), \sum_{n \neq i} s_n(t)).$$

This made WSJ0-2Mix part of a broader research program on unknown and varying source counts (Takahashi et al., 2019).

3. Performance trajectory in monaural separation

Reported results on WSJ0-2Mix show a long progression from early PIT and deep clustering baselines to increasingly strong time-domain and Transformer-based separators. Early PIT results reported 8.4 dB SDRi for PIT-CNN, with DPCL at 6.5 dB SDR improvement under optimal assignment and DPCL+ at 10.3 dB under default assignment, while an oracle IRM reached 12.3 dB (Yu et al., 2016). Recursive OR-PIT later reported 14.8 dB SI-SNRi and 15.0 dB SDRi with a single model handling multiple speaker counts (Takahashi et al., 2019).

The time-domain masking era pushed the benchmark substantially further. SepFormer, a dual-path Transformer separator, reported 22.3 dB SI-SNRi and 22.4 dB SDRi on WSJ0-2Mix with dynamic mixing, establishing a strong Transformer baseline (Subakan et al., 2020). Sandglasset emphasized multi-granularity contextual modeling and achieved 20.8 dB SI-SNRi and 21.0 dB SDRi with 2.3M parameters, rising to 21.0 dB SI-SNRi and 21.2 dB SDRi after post-training with same-speaker mixture augmentation (Lam et al., 2021). MSFFT-Net extended the dual-path idea with multi-scale feature fusion and reported 21.0 dB SI-SNRi and 21.2 dB SDRi for MSFFT-2P without data augmentation (Xu et al., 2022).

Resource-efficiency became a parallel concern. RE-SepFormer replaced overlapping latent chunks with non-overlapping blocks and compact latent summaries, reporting 18.6 dB SI-SNRi and 18.9 dB SDRi with 8.0M parameters and 6.3 GMACs/s, versus 25.7M parameters and 69.6 GMACs/s for the full SepFormer reference configuration (Libera et al., 2022). A related investigation of efficient self-attention mechanisms found that Reformer-based attention could outperform Conv-TasNet on WSJ0-2Mix while being faster at inference and comparable in memory consumption, even though full attention remained best for absolute separation quality (Subakan et al., 2022).

More recent architectures approached the reported upper bound. MossFormer combined gated single-head attention, convolution-augmented joint local and global self-attention, and dynamic mixing, reaching 22.8 dB SI-SDRi on WSJ0-2Mix, which was reported as only 0.3 dB below an upper bound of 23.1 dB (Zhao et al., 2023).

System	Reported WSJ0-2Mix result	Notable emphasis
PIT-CNN	8.4 dB SDRi	Early PIT benchmark
OR-PIT	14.8 dB SI-SNRi, 15.0 dB SDRi	Unknown speaker count
Sandglasset (MG) + PT	21.0 dB SI-SNRi, 21.2 dB SDRi	2.3M-parameter multi-granularity model
SepFormer + DM	22.3 dB SI-SNRi, 22.4 dB SDRi	Dual-path Transformer baseline
MossFormer (L) + DM	22.8 dB SI-SDRi	Near reported upper bound

This progression suggests that WSJ0-2Mix has served not only as a scorekeeping benchmark but also as a controlled environment for comparing architectural inductive biases: recurrent, convolutional, dual-path, multi-scale, efficient-attention, and fully Transformer-based designs.

4. Extensions to target extraction and speech recognition

WSJ0-2Mix has also become a standard benchmark for target speaker extraction (TSE), where the goal is not to separate all speakers but to extract the one indicated by an enrollment utterance or speaker cue. A SepFormer-based TSE system using sparse LDA-transformed speaker embeddings reported up to 9.9% relative improvement in SI-SDRi, with a best system at 19.4 dB SI-SDRi and PESQ 3.78 on WSJ0-2Mix (Liu et al., 2023). X-SepFormer reformulated TSE training around speaker confusion at the chunk level and reported 19.4 dB SI-SDRi, PESQ 3.81, and a 14.8% relative reduction in speaker confusion errors (Liu et al., 2023).

The benchmark also supports target-speaker ASR and multi-speaker ASR. CONF-TSASR, a Conformer-based masking-plus-ASR model with a scale-invariant spectrogram reconstruction loss, reported 4.2% target-speaker WER on WSJ0-2mix-extr (Zhang et al., 2023). SQ-Whisper adapted Whisper to overlapping-speech recognition through trainable speaker-query prompts and reported 5.5% WER on WSJ0-2Mix, reduced to 4.4% with speed perturbation (Guo et al., 2024). For multi-speaker ASR without a target enrollment, a conditional Conformer-CTC model reported 19.9% WER on WSJ0-2mix with variable-speaker-count training, and a jointly optimized separation-plus-ASR system reported a specialized two-talker WER of 7.5% after fine-tuning both front end and recognizer (Guo et al., 2021, Neumann et al., 2020).

Earlier ASR work used WSJ0-2Mix primarily as a preprocessing benchmark for separation-plus-recognition pipelines. A deep clustering front end with a DNN-HMM recognizer reached 16.5% WER, which that study described as the best performance reported thus far for the dataset at the time (Menne et al., 2019). A plausible implication is that WSJ0-2Mix has gradually shifted from being a pure separation benchmark to a broader evaluation substrate for speech-conditioned extraction and overlapped-speech recognition.

5. Multichannel, spatial, and variable-source extensions

Although originally dominant as a monaural benchmark, WSJ0-2Mix has been extended to multichannel and spatialized settings. SDNet is a dual-channel time-domain model that infers speaker and direction representations sequentially, using inter-channel attention correlation

$\text{IAC} = \text{softmax}(E_1E_2^T)$

to encode spatial cues. On dual-channel anechoic WSJ0-2Mix it reported 25.31 dB SDRi and 25.71 dB SiSNRi, while under reverberant settings it reported 10.64 dB SDRi (Li et al., 2021).

Unknown speaker count is another major extension. OR-PIT jointly trained on WSJ0-2mix and WSJ0-3mix used a single recursive model to separate one speaker at a time and could separate four-speaker mixtures never seen during training (Takahashi et al., 2019). A joint source counting, separation, and ASR system similarly combined iterative extraction with source-counting mechanisms and reported promising counting accuracy, source separation, and recognition on simulated clean mixtures from WSJ0-2mix and WSJ0-3mix, while generalizing to WSJ0-4mix (Neumann et al., 2020).

These variants change the meaning of success on WSJ0-2Mix. In monaural fixed-output separation, the central question is output assignment and waveform fidelity. In dual-channel and recursive formulations, the same benchmark family is used to probe spatial inference, stop criteria, dynamic output selection, and source-count estimation. This suggests that WSJ0-2Mix persists partly because its controlled construction makes it easy to layer new experimental assumptions onto an already familiar task.

6. Limitations, controversies, and successor benchmarks

Despite its status, WSJ0-2Mix has significant limitations. A broad critique identifies limited speaker and vocabulary diversity, repeated utterance usage, training and validation splits that share speakers, 100% overlap that differs from real conversational conditions, and the proprietary status of the underlying data (Cosentino et al., 2020). A related ASR study argues that in realistic scenarios speech overlaps occur only in segments, and shows that applying deep clustering naively to sparsely overlapping speech can damage single-speaker regions, making segmentation and selective application important (Menne et al., 2019).

Generalization across datasets is a recurring concern. One cross-dataset study reports that models trained on WSJ0-2Mix or WHAM! show a 4 dB SI-SDR drop on LibriMix and around 3–4 dB worse performance on VCTK-2mix relative to LibriMix-trained models, while LibriMix-trained models generalize back to WHAM! with only a 0.8 dB SI-SDR drop (Cosentino et al., 2020). This motivated LibriMix as an open-source alternative designed for broader diversity and improved cross-dataset generalization (Cosentino et al., 2020).

A more recent controversy concerns the references themselves. An analysis of SI-SDR with noisy references argues that WSJ0-2Mix reference signals are not truly clean, that “the SI-SDR is bounded by the reference SNR in the case of ideal estimates,” and that maximizing SI-SDR with noisy references can encourage models to reproduce reference noise rather than remove it. That study reports a negative correlation between SI-SDR and perceived noisiness across models on WSJ0-2Mix and Libri2Mix, and uses enhanced references plus NISQA.v2 evaluation to show that cleaner outputs can receive worse SI-SDR when the benchmark references remain noisy (Jepsen et al., 20 Aug 2025).

Taken together, these criticisms do not nullify the benchmark’s historical value. They indicate instead that WSJ0-2Mix is best understood as a highly influential controlled benchmark whose conclusions are strongest for comparative algorithmics under matched conditions, and weaker when extrapolated to real-world overlap patterns, perceptual quality, or out-of-domain robustness.